Articles

Are "FROM scratch" images incompatible with vulnerability scans?

2024-10-04T00:00:00+00:00

I decided to invoke Betteridge's law of headlines</a> to start the article off on a good note.</p>

No, FROM scratch</code> Docker images are not incompatible with vulnerability scans, but it requires a bit of effort to make them work.</p>

Huh? Vulnerability scanning?

#</a></h2>
A bit of context on the whole ordeal. Modern software development is riddled with tons of dependencies. In the good old days, all software dependended only on other packages published for the same distribution, and all packages were separately packaged for each distribution. It was a rule that no bundled packages were allowed, so if you upgraded the installed version of some library on your system, it would affect all the applications that depended on it. Then came the the different package managers separately for programming languages – probably Perl's CPAN</a> was the one of the first ones, but then similarily for Python, Ruby, and so on. And finally Node.js broke the camel's back with the proliferation of node_modules</code> directories in every project, each containing multiple versions of the same library.</p>
All these dependencies obviously can have security vulnerabilities, so it becomes important to track which dependencies are included with each piece of software. This can no longer be done simply by looking at date or version of the distribution, but instead a full scan of the entire software must be made to locate all bundled dependencies. This is the job of vulnerability scanners, of which Trivy</a> is a fine example. These tools take a piece of software, or a Docker image, and build a comprehensive list of all included dependencies, and then compare them against published vulnerabilities.</p>
Since dependencies these days come from both the distribution and the programming language package managers, the scanners generally detect the OS of the image and then check the installed packages based on that, and in addition scan for any language-specific package manager state files, such as package-lock.json</code> or Cargo.lock</code>, and detect software dependencies based on those. So far so good.</p>
The itch with FROM scratch</code>#</a></h2>
Docker images traditionally used to have the entire build history of the software included in its layers. All the build tools, all the source code, all the packages. This was a treasure trove for vulnerability scanners as they had all the data to work with. But then people realized that it's a bit of a security vulnerability in itself to have all the tools included in the image that are not needed at runtime, and the images are unnecessarily bloated. So people started turning to multi-stage Docker builds, where the build stages have the tools but the final image has only the files required at runtime.</p>
The evolution of this, bit by bit, is FROM scratch</code> images, where the final stage of the build starts from an empty image, and only the necessary binaries and libraries are copied onto it. This is especially useful for compiled languages, though sometimes done for interpreted languages as well. The resulting images are as small as possible (even smaller than "distroless" images) and are difficult to exploit in case of a vulnerability since there are no extra tools to exploit.</p>
But this is where the problem arises for vulnerability scanners. There's no installed operating system, so they can't know what vulnerable software might've been there when compiling the software. And there's no package manager state files, so they can't know what dependencies were included in the software. Hence, most vulnerability scanners will just refuse to scan FROM scratch</code> images entirely, which can be a showstopper in heavily regulated environments.</p>
Scratching the itch#</a></h2>
So, the first step to make FROM scratch</code> images scannable is to include the language-specific package manager state files in the image. This means simply putting your package-lock.json</code> or Cargo.lock</code> in the image next to your software, even if nothing actually requires it. The security scanners scan all files in the image, so they'll usually just pick it up where ever it is placed. This part is easy, but also not necessarily sufficient to track everything that might actually cause an exploitable vulnerability. Even if the language-specific package manager state file is included, it doesn't mean that everything that is included in the software only comes from there. Any system libraries, such as openssl</code> or libz</code>, are not necessarily versioned in the lock file, so vulnerabilities for them might not be tracked, yet they might be statically linked or otherwise included with the software.</p>
But even more disturbingly, it seems that just including the programming language package manager state file is often not enough to get the image scannable. Many vulnerability scanners want to first detect the OS of the image and will determine based on that if they support the image or not. If they don't detect the OS, or do not support that OS, then they refuse to also scan for the language-specific package manager state files. This is certainly a misfeature of the scanners, so it's likely to get fixed in the future, but we can't wait for that.</p>
The solution is to fake the image to look like a supported OS image</strong> by including the release files that the vulnerability scanner is looking for, as well as the package state database for the image. Ideally, you should use these files directly from the image you are compiling the software on, as then it will track all the dependencies of any package that might've contributed to your binary. However, in practice, the compiler image might contain a ton of irrelevant packages, which means there's a ton of vulnerabilities reported that are not relevant for your software. The good solution to this is to minimize your compiler image to only contain necessary packages, but simply truncating the package manager state file to a single non-vulnerable package also works, if you just want the image to be scannable and don't care about any system libraries.</p>
Yap yap yap, just show me how it's done!#</a></h2>
For Alpine Linux:</p>
FROM</span> alpine:latest</span> AS</span> builder</span>
</span>
</span>#</span> Build your software here
</span></span>
</span>FROM</span> scratch
</span>
</span>COPY</span> --from=builder</span> /build/Cargo.lock /
</span>COPY</span> --from=builder</span> /build/target/release/mybinary /
</span>COPY</span> --from=builder</span> /etc/alpine-release /etc/
</span>COPY</span> --from=builder</span> /etc/os-release /etc/
</span>COPY</span> --from=builder</span> /lib/apk/db/installed /lib/apk/db/
</span></code></pre>
For Debian:</p>
FROM</span> debian:latest</span> AS</span> builder</span>
</span>
</span>#</span> Build your software here
</span></span>
</span>FROM</span> scratch
</span>
</span>COPY</span> --from=builder</span> /build/Cargo.lock /
</span>COPY</span> --from=builder</span> /build/target/release/mybinary /
</span>COPY</span> --from=builder</span> /etc/debian_version /etc/
</span>COPY</span> --from=builder</span> /etc/os-release /etc/
</span>COPY</span> --from=builder</span> /var/lib/dpkg/status /var/lib/dpkg/
</span></code></pre>
Figuring out the right commands for a distribution of your choice should be quite straightforward. And that'd be it.</p>
In conclusion#</a></h2>
This approach seems to work with at least AWS Inspector and Trivy, but probably works with others as well. They are not so complex beasts, after all.</p>
As FROM scratch</code> images become more popular, it's likely that the need for such workarounds will diminish, but for now, this is the way to go. Also, hopefully tools like cargo-auditable</code></a> will remove the need to include Cargo.lock</code> separately as the same (and better) information is included directly in the produced binaries.</p>

Redis + Strong consistency = AWS MemoryDB 2024-08-16T00:00:00+00:00 </p> Redis</a> (or the fully open source Valkey</a>) is a very versatile and fast in-memory data store. Read operations take microseconds, and write operations generally take 1 millisecond. It started as a very simple key-value database, but it has picked a huge number of commands and features over the years. There's plenty of different datatypes, including streams</a> and JSON</a>. There's also embedded functions written in Lua</a>, which are fully transactionally isolated, so it's possible to do complex transactional operations with the minimum of data transfer. It's no longer just a cache, it's a full blown database that is suitable for a wide variety of uses. In short, Redis is an extremely useful tool for any fast state you might require that cannot be locally stored in memory.</p> The biggest drawback for using Redis for anything else than a simple in-memory cache is that it doesn't provide strong consistency. Under normal operation, when a master is stable, Redis is very consistent as everything happens just in-memory and it is single threaded. But everything being in-memory means that Redis will acknowledge successful writes before they have been persisted anywhere, which means that if the master goes down, you might lose just written data. Redis can be configured with a number of different options for persistence, but most commonly it writes to an AOF (append-only-file) once per second.</p> </p> To combat this, there is the WAIT</code> command which can be used to wait until a certain number of replicas have acknowledged the write, which means that at least it's not up to just a single machine anymore. This doesn't mean that it would be synchronized to disk necessarily (for example if all replicas lose power / crash at the same time), but if they are independent enough it's quite likely that at least one replica will persist the data to disk. But even then there are no actual guarantees that a replica being promoted would have all the data. It tries to promote the replica with the most up-to-date state, but given that failure scenarios are often quite complex, it's still just best-effort consistency. In the worst case, a replica could be promoted that doesn't have any</em> data, so the whole database would start from scratch.</p> </p> But what if I told you there's an alternative (if you are on AWS)...</p> AWS MemoryDB#</a></h2> MemoryDB is a Redis OSS compatible database that claims to offer both durability and strong consistency, while keeping the fast in-memory performance of Redis. That sounds almost too good to be true. So how do they do it? Is it really compatible with Redis? What are the guarantees they provide?</em></p> Consistency#</a></h3> What MemoryDB promises is that the moment a write operation completes, that write operation is durably persisted (eleven nines, just like S3) and strongly consistent, meaning that no operation will be run on the database without that change present. This means it acts like any ACID database with SERIALIZABLE isolation level - that's everything one could hope for. This is exactly the same as Redis offers by default if the master is stable. It's still single threaded.</p> Speed#</a></h3> But it can't do that while keeping the exact same speed that Redis has, can it?</em> Well, no, obviously it can't, since Redis answers immediately after something has been written to memory only on a single machine. But it can get surprisingly close.</p> First of all, read operations are not affected at all and exhibit the same performance as with Redis, as is to be expected. Well, to be exact, it's even a bit better than Redis, but that's due to Amazon optimizing the database for their hardware with some enhanced IO multiplexing. So the response times for both databases are in microseconds - this is an in-memory database after all.</p> Write operations in Redis will complete usually in 1-2 milliseconds. For MemoryDB that same figure is 3-5 milliseconds. That's obviously slower, and a meaningful difference, but still quite a bit faster than what most databases can manage. It's pretty much on par with a normal write + WAIT</code> for Redis. Especially if there's any network latency in general, a couple more milliseconds will end up fully hidden in that latency.</p> This figure is also pretty much exactly the optimal result one can theoretically achieve, while keeping the same availability approach. In order for the data to be durably persisted, it needs to be stored in multiple data centers. In AWS this means multiple availability zones inside a region (an availability zone might actually have multiple data centers, close by, as they are huge). These availability zones are physically located at least a few kilometers away from each other and a maximum of 100 kilometers. The networking round-trip latency is less than 2 milliseconds, but can't be instant, because the speed of light isn't instant. Hence if Redis takes 1 millisecond to process the operation, and getting the same operation delivered to two other datacenters takes a 2 millisecond round-trip, we arrive exactly to the 3 millisecond minimum time.</p> </p> Implementation#</a></h3> So how do they do it?</em> They add a strongly consistent transactional log into the normal Redis codebase, where the implementation of the log is AWS secret sauce. The transactional log is placed after the normal in-memory operation for Redis, so once the operation is done, the results of the operation are committed into the log and will end up getting relayed to the replicas. And each replica will always ensure it has the entire log replayed before it can become a master.</p> But if this were done naively, then every write operation would end up taking three times as long because of the transactional log, which would reduce the throughput of the entire database similarily. But that doesn't seem to be the case as total throughput, while less than a standard Redis installation, is quite close in the end. That can't be the way they do it, then.</em></p> What they actually do is that they only delay the sending of the acknowledgement message after a command until they have received an acknowledgement from the transactional log. This means that the database itself can actually perform operations at full speed in the in-memory database, and only the clients are delayed until there is confirmation that the data has been durably persisted and will be strongly consistent. This is quite a cool idea, and it's easy to reason about the performance knowing this.</p> </p> How do I know all this?</em> AWS published an actual research paper on MemoryDB</a> where they explain all this. I'm just making it worse by explaining it in my own words.</p> Compatibility#</a></h3> Everything else about Redis stays the same, as everything is first executed in-memory and then written to the transaction log. So we know it supports all the same stuff as Redis does without having to reimplement everything and keeping all the quirks that we've already gotten used to (unliked some... krhm... DocumentDB</a>). And they've added the same JSON support that's currently part of the not-so-free Redis offering.</p> Replication in Redis is also a very standard feature and thoroughly battle tested. This solution just improves the behaviour around the replication stream, but the actual data elements are the very same. So we don't have to worry about new quirks around randomness or functions or other similar things appearing, but those will still be just like they are in Redis.</p> Cost#</a></h3> AWS offers the standard Redis as well, under the name Amazon ElastiCache with Redis OSS compatibility. This makes it easy to compare the prices of that against MemoryDB. Comparing the costs for on-demand instances, they seem to be charging around 40-50% more for the same instance types. Given that Redis costs are usually quite low compared to the value they bring, that isn't too bad.</p> In addition to that, there's a cost of $0.20 / GB for data written. This is obviously to cover the cost of the transactional log. Usually Redis is read heavy, and the written data elements are quite small. Evaluate your use case thoroughly, but I would wager a guess that this cost is also not prohibitively expensive.</p> Summary#</a></h3> Overall, this gives a very interesting new choice for development. Usually anything that requires strong consistency has had to have been implemented with DynamoDB, or some other real database. And that has a cost per transaction that may get significant. Anything read heavy, or more ephemeral, would then be handled by Redis where the cost would simply be based on the dataset size (the memory size of the nodes).</p> MemoryDB allows us to get pricing that's based on the dataset size, and not the amount of operations, for a strongly consistent database. A database that is in-memory speed for reads and nearly in-memory speed for writes. I'm betting that's quite an interesting proposition for a number of developers.</p> Now if they would just add cross-region replication...</p>

Cost
Summary

Summary

Articles

Are "FROM scratch" images incompatible with vulnerability scans?

In conclusion
`cargo-auditable</code></a> will remove the need to include Cargo.lock</code> separately as the same (and better) information is included directly in the produced binaries.</p>`

Redis + Strong consistency = AWS MemoryDB