The Invisible Things

Debunking the myth of the dark Web

Mon, 13 Nov 2023 00:00:00 +0000

This is a blog post I found while exploring the dark Web a long time ago, and it caught my eye after reading a few paragraphs. The author, whose name is Alice, has taken a rigorous attitude, devoting 7,000 words to the ins and outs of the dark web, and the examples cited in the article are all footnotes (up to 44), which shows that the author has spent a lot of energy, and has a very professional level, and is completely writing a serious paper. In order to read the convenience, I deleted the footnotes, readers if there is a deep interest, you can see my personal website blog, I am afraid to add a link to the article will be blocked… (website to see the signature should be written, the site for two years, the number of page views but I ~~ hahahaha).

First of all, write some of their own reading, the article is very long, readers need to calm down to straighten out their ideas, the article likes to drop the Internet book bag, involving rich knowledge points, many knowledge points are not available to the average person, but also gave the footnotes, if seriously reading, it is estimated that there is no certain level to spend some time to see the footnotes. Such an article is much smarter than a cool article with a few gory and scary photos to create a scary atmosphere of the dark web. What I admire more is the author’s ability and attitude, as well as the feelings full of idealists.

The article made me lament that there was already a blueprint for anonymous electronic money transactions in 1981, and until now Bitcoin has only been a tool for human evil (money laundering/black market/manipulation). In fact, even the original idealist free/equal Internet is very different from today’s centralized/monopolistic Internet. It’s ironic that what started out as a well-intentioned tool to defend everyone’s privacy has instead been used by the evil of humanity to become a monster that everyone is Shouting at.

It is even more ironic that on the open platform of the Internet, the material about the dark web is so incendiary and arouses emotions that most people have little understanding of it. (Most people’s understanding of the virus is also the same) and on the so-called “dark web” this private platform, it is rigorous and objective to describe its past life.

The same is true of human history, where the few deceive and control the many; There is also an idealistic minority who want the majority not to be deceived and controlled. So far history has repeated itself, but I believe it will have a different future.

With the full text –

Debunking the myth of the dark Web

Author: Alice

Published: November 17, 2017

Technology pioneers’ quest for Internet freedom began shortly after the Internet was born. In 1981, cryptographer David Chaum designed an anonymous E-mail system, and a few years later outlined a blueprint for an anonymous electronic money exchange system in a paper declaring that it “would render Big Brother obsolete.” Three years later, TimothyC. May argued that just as printing weakened and changed the medieval guilds and social power structures, cryptography would do the same. A small wire fence made it possible to build vast farms and ranches in the Western United States, and the small branch of mathematics known as cryptography became a sharp pair of scissors to strip away the wire fence surrounding “intellectual property.” In 1993, Berkeley scholar Eric Hughes further supplemented the basic idea: We cannot expect governments, corporations, or other faceless giant institutions to benevolently grant us privacy, and if we expect privacy at all, we must stand up and defend it.

The anonymous web technology we see today is the result of the efforts of these hackers. Unfortunately, their years of hard work have been seriously distorted in the past few years, and they have become what the media calls the “dark web.” While the Silk Road is partly to blame for this, it is largely to blame for the wild urban legends that, because of their dramatic and mysterious nature, are widely disseminated on social networks and half-understood subject media. They will tell you that the entire Internet as we know it is only 5% of it, and that more than 95% of the content is on the “deep web” and the “dark web.” So who’s involved? There are alleged secret societies, mafias, human experimentation, illegal drugs, human trafficking, and hidden conspiracies. Today darknet, tomorrow darkweb, then deepweb, in short, all kinds of buzzword, and “Internet security” is not a word.

These myths have serious implications not only for anonymous technologies and their communities, but also for Internet privacy and security itself. In order to dispel such myths, it is necessary to conduct a simple research on the sources of these claims, so that history can be used as a mirror to recognize the facts, and over-exaggerated conspiracy theories can be destroyed. Finally, we need to recognize the misleading and harmful nature of the term “dark web” and recommend that we stop using the term “dark web” and use other terms, such as “anonymous web.”

Due to the short writing time, the article is not only very long and lax sorting, I hope readers can understand.

Deepweb

Before dealing with the confusing definition of the “dark web,” let’s first talk about the uncontroversial “deep web.”

“Deep Web” is a term used in web search to refer to content that exists on the Internet but is not indexed by search engines. The simplest examples are websites that require a login to view, public databases that require a form to query, and so on. Many studies believe that the scale of the data that has not been included is much larger than the scale of the network data that can be searched, and the public data is clearly on the network, but can not be found by search engines, so it is a major problem that search engines need to solve. Especially in the early days of Internet development, when search engines were not perfect, relevant reports appeared almost every few years. There are also academic teams that have built tools to help users query deep Web data, which is where the claim that only 5% of the Internet we see comes from. For example, this report by ABC Australia in March 2010 even used “iceberg” as an example, which foreshadowed the legend in the future.

Early meaning of Darknet

The term “darknet” had several early meanings, none of which have anything to do with today’s urban legend usage. The first meaning, which is closer to its literal meaning, refers to an address space on the Internet that has assigned IP segments and routes, but does not actively run any network services or servers. Therefore, any packet entering this IP address space seems to be entering a black hole, hence the name “dark web”. Such networks can act as a kind of honeypot, deploying logging and intrusion detection systems to track malicious traffic on the Internet, such as network viruses, DDoS attacks, etc., because any data attempted to send to the “dark web” is inherently problematic. The term appears in several papers on network architecture and security research, such as this MIT paper, and the team can be reached at darknet@mit.edu…

The second, more interesting implication, comes from a paper published in 2002, which suggests that the “dark web” is not a separate physical network, but rather a layer of applications and protocols on top of the network, such as “P2P file sharing, CD and DVD copying, and keys and passwords shared in email and newsgroups.” The reason for this alarmist description is that the paper, written by Microsoft developer PeterBiddle for digital rights management technology, argues that such private web sharing poses a serious challenge to DRM, which they do not want to see. Around the same time, groups led by the RIAA and the MPAA began a harsh crackdown on piracy in the name of protecting artists, and many American Internet users had a lot to say about it. In 2005, journalist J. D. Lasica published a book titled Darknet: Hollywood’s War Against the Digital Generation, a book that describes Hollywood’s copyright wars, the open Culture movement, etc., is also the meaning of the “dark web”. At the same time, the Freenet project, which has been active since the beginning of this century, also released a new version of v0.7 in 2007, adding a “dark web” mode, meaning that Freenet will only be connected to the trusted friend node for data exchange, so as to improve the privacy of user communication and avoid interference from outsiders. Here, the “dark web” is the same definition.

The third meaning refers to the network used for organized crime. This usage was first seen in the special research on online extremism and terrorism of Arizona State University, which used web crawler, statistics, natural language processing and artificial intelligence technologies to conduct a large amount of data analysis on the network promoting terrorism, and studied the operation mode and communication characteristics of such communities. In a 2003 slide study, the “dark web” was defined as the other side of the Internet, used by terrorist and extremist groups for propaganda. In 2006, a feature report on the study was picked up by NPR, Fox News and others, probably the first time a news outlet used the dark Web. But it’s worth noting that the “dark web” here doesn’t have any meaning of “anonymous,” “encrypted,” or “secret,” and that the terrorist websites in question are all accessible directly from a browser (the slideshow begins with sites that promote extreme racism, such as Stormfront, as an example), and that Tor wasn’t mentioned until recently. As a result, the term “dark web,” which is the most widely abused term in today’s sense and pervades media reports and urban legends, still does not exist.

It was not until November 2009 that The Guardian reported on The dark side of the internet, which briefly covered the Pirate Bay, the development of Freenet, and the existence of illegal content on the web. The Reddit forum’s /r/darknet/ section was created around the same time, apparently in response to the news. However, there is only a brief mention of Tor and links it to the “dark web”, saying that it is a Freenet-like system, but there is no mention of Tor’s Hidden Service features, and the time is set for all of 2010. Typing Darknet or Darkweb still yields no results, except for a puzzle game of the same name and something like a “Dark Web Color Scheme Guide.” A search of Tor Hidden Service, however, turns up some technical data. It can be seen that until 2011, the anonymous website technology itself, such as Tor, and the various shady websites hosted by it were still not associated with the term “dark web”, and its modern usage was still not formed.

Silk Road

The turning point came in February 2011, when Silk Road launched after six months of development. Just four months later, tech media outlets like Gawker and Motherboard were the first to report on the black market for drugs operating on Tor. However, the words “deep” and “dark” did not appear in the news at all, only the role played by Tor was mentioned, and then more mainstream media followed up, and Congressman Charleschumer’s attention called on law enforcement agencies to act. But in these early articles, there was no mention of the “dark web” or “deep web,” instead focusing on Bitcoin.

Meanwhile, in April, Anonymous members launched the #OpDarknet initiative, claiming that one of their IRC servers had begun blocking proxy servers, Tor, and VPNS, and called on Anonymous members to use i2p. This is probably the earliest modern use of Darknet. In October of the same year, Anonymous launched an attack on the child pornography site Lolita City on Tor and the Freedom Hosting service, exposing a large amount of user data, and again using the #OpDarknet banner, claiming to seek justice. Then, in reports in the Huffington Post and The Wall Street Journal, there was a natural interpretation of one of the terms “Darknet,” which said it “protects dissidents from government crackdowns but also serves as a venue for everything from drugs to child pornography.” According to the former, “The sites on Darknet are part of the Invisible Web, sometimes called the Deep Web, which includes content other than the Surface Web that is indexed by search engines.”

Over the course of the year, some individual blogs began to refer to the Tor HiddenService as “deep web” and “dark Web”, such as using Anonymous’s Operation Darknet captions. At this point, the term “dark web” was first associated with these anonymous sites on Tor and confused with “deep web,” a usage that would continue to spread. Soon after, the famous image of the iceberg appeared, and the earliest documented appearance of the image was this version uploaded to Imgur on May 31, 2011, which has been viewed more than 150,000 times. It compares the Internet to an “iceberg” and places easily accessible sites above the water, while lesser-known sites that require technical tools are placed below, and asks if any of its readers have reached the deepest level of the water. The image is poorly reworked and in poor quality, and it also suggests that niche forums such as 4chan, 808chan, 7chan and 420chan are the dividing lines above and below the water. It stands to reason that the image first appeared on a discussion post on 4chan before it was uploaded to Imgur. Full of 4chan’s usual color of abuse and parody, it soon aroused the attention of unknown outsiders, leaked out of 4chan, and was posted elsewhere.

In one Reddit post, for example, the user asked /r/askreddit what the image meant and got 161 votes. It was around this time that the standard “deep web” and “dark web” myth that “the entire Internet is only 5% known of it, and that more than 95% of the content is located on the deep web and dark web, and that there are” government secrets, secret societies, mafias, human experimentation, illegal drugs, human trafficking “claims began to emerge. The most prominent example is the “Mariana Net,” which was posted around the web in June 2011. It takes its name from the Pacific Ocean’s Mariana Trench, which refers to the deepest and most secret corner of the so-called “dark Web,” and is the subject of many urban legends. This statement is likely to have been directly influenced by the “iceberg” image.

By the end of 2011, such claims were widespread. We don’t find the original urban legend text, probably because it originated on 4chan and other imageboards and is no longer recorded, but references to it can be found everywhere, such as in the K-3 BBS of Taiwan villagers, where the discussion is very conspiracy-tinged; In this Channel 4 news report on September 1, the word “Darkweb” also appeared; Reddit’s /r/deepweb/ section was created in June 2011; By 2012, “Darkweb” had been frequently seen in mainstream media, such as this BBC report about illegal drugs on the “dark web”; A book called Deep Dark Web was published the same year; After the FBI shut down Silk Road and arrested those involved in the case in 2013, public opinion reached its climax, eventually forming the current Tor = “dark web” situation.

In short, we can see that the “dark web” had almost no use before 2010. After that, Anonymous’ #OpDarknet was the first to bring the term to public use. With the launch of the Silk Road that same year, the media quickly confused the meaning of “deep web” and “dark web.” At the same time, 4chan users also naturally noticed these phenomena, and produced “iceberg” pictures to discuss and joke. As a result, all the media were misled by 4chan’s “masterpieces”, so that the “deep web iceberg” theory entered the official news report. During this time, various rather dramatic urban legends began to spread, such as the “Mariana Net,” which eventually became the nonsense we read today, such as this one.

Anyone browsing foreign urban legends on Reddit is no doubt familiar with the term “deep web.” It’s a domain that lurks beneath the web as we know it, that you can’t find through a search engine or a web site, that’s full of all kinds of unimaginable illegal activities, like drug smuggling, arms dealing, hacking, money laundering, human trafficking, you can even buy murder through the dark Web, join a cult, buy and sell organs, Or livestreaming perverted killings or watching child pornography. Even more striking is that the content of the dark web accounts for 94% of the entire web, in other words, the pages that we can normally access, only account for 6% of the entire web. The hacker successfully hacked into the “dark web” full of evil activities, and witnessed the extreme torture images that left him with a lifetime of shadow.

If the general domain is an iceberg, then the dark web is the bottomless ice beneath it. To access The dark web, you have to go through a system called Tor, a technology originally developed by American military scientists that allows all users to remain anonymous on the Internet, also known as The Onion Router because the passwords that protect data are layered on top of each other like an onion. Since such technology has fallen into the hands of people with a heart, it has become a dark world, if you want to also through the Internet to provide teaching into the dark web, but those who enter the dark Web hackers definitely have a way to find your server in the first time, even your IP address, causing a threat to life.

Stop using the term “dark web.

Thus, the “dark web” is an ambiguous and heavily mixed term that has not only lost any definite meaning in itself, but has also become associated with various urban legends. First, the term “dark web” is riddled with factual errors, such as “astonisingly, the dark Web accounts for 94% of the web’s content”, in order to attract readers’ attention, without mentioning the true size of Tor Hidden Service. The OnionScan project scanned the entire Tor network in 2016 and found a total of about 30,000 sites, and with Freedom Host II going offline, the number of Onion sites has reached an all-time low of about 4,000 (if I were a media outlet, I’d write this: Shock! The Dark Web has become 80% darker! ). In addition, according to the statistics of Tor Metrics, the current Tor network can exceed 100 Gbps of traffic, but Hidden Service traffic only accounts for 1.5 Gbps, about 1% of the entire network traffic, most of the traffic is almost through the exit node browsing the World Wide Web. Instead of being “94% of the web,” building websites and services is one of the community’s top priorities. For lurid dramatic effect, these urban legends also exaggerate the violent elements of the Internet, and while it is true that some vicious outlaws use Tor to cover themselves, the extent of it is exaggerated, and much of it is completely false, like the “invisible dark world”. If you want to, you can also access the dark Web through the education provided on the Internet, but those who enter the dark Web definitely have a way to find your server in the first time, even your IP address, causing a threat to life “, as if I open two pages with Tor Browser, I will be killed? Are you sure you haven’t been watching too much Hell Girl? Even the characters didn’t die that fast… Moreover, there is a huge amount of malfeasant activity happening every day on the open Internet, at least at the same level, if not as serious, as on the so-called dark Web; At the same time, these urban legends resort to mystery and conspiracy, and not only do not play an applied role in popularizing cybersecurity knowledge, but also mystify security technology.

This leads directly to serious consequences. For example, the British police recently said that visiting the so-called “dark web” indicates that you may be a terrorist, and encourages citizens to actively report to the police, in case someone sees Tor Browser in the future, someone will call the police. Second, it seriously misleads newcomers to treat anonymous web technologies with a distorted mindset, and especially to use them in a curious manner, rather than as a practical security tool, always trying to dig something out of the ground. Tor developers in a speech last year, for example, one after the researchers see https://facebookcorewwwi.onion/ was amazed at the meeting asked him: “what’s the matter? Why did I find a dark Facebook on the dark Web?” Ironically, this version of Facebook allows activists in authoritarian countries to speak out without revealing their geographic location. Readers who read the comments on Alice’s blog, saying that they were “shocked to see the ‘dark web’ with Chinese characters for the first time,” may have been misled. Finally, this has a negative impact on the developers of anonymous web technologies and the community as a whole, meaning that exaggerating urban legends will encourage more and more people to set up websites dedicated to publishing content related to urban legends, rather than providing innovative, practical and safe web services. This image will become self-reinforcing and will only lead to the deterioration of the entire online community environment. Internet privacy cannot be promoted.

Commenting on a browser plugin called Darkweb Everywhere, Tor core developer Roger Dingledine blasted it:… I can’t accept the name: I don’t want to promote such dark images of icebergs at all.” The term “dark web” only exaggerates and reinforces the FUD that HiddenService is facing. We’ve been debating these terms lately, and I now tend to favor The term “The PrivateWeb.” While it conflates the security features that users get from Tor with the security features that websites get, both are really necessary, so I have no problem with the name; The Tor team’s speech at the 2015 CCC conference also mentioned that people need to realize that Tor is just another network protocol. If we need to transfer plaintext web pages, we use HTTP; If we need secure delivery of web pages, we use HTTPS, if we need self-authenticated, end-to-end encrypted anonymous delivery, we use Onion, and that’s it. Here we urge readers to use the term “privacy network” or “anonymous network” (Alice prefers the latter), and stop using the term “dark web” to call these web technologies! The entire anonymous web technology community would appreciate it if you could practice this for yourself.

Finally, having spent a lot of time talking about urban legends and horror stories that the news media talk about, let me also briefly tell you the stories that they don’t tell you: In 2006, a whistleblower published the improper actions of a pharmaceutical company. As a result, under the pressure of lawyers in the name of “copyright”, the domain name of the published document was stopped for many times, and finally it was successfully published on Tor HiddenService. This was one of the first examples of Tor Hidden Service providing citizens with freedom of speech protection, and later wikileaks did the same; Today, thanks to the efforts of the Tor community, platforms like GlobaLeaks and SecureDrop have been joined by more than 30 news outlets, making it easy for any citizen to submit raw material, such as political corruption, to multiple news outlets. According to these outlets, they have received some key clues from here (what exactly? Media: Don’t answer, don’t answer, don’t answer); In addition, Hidden Service has a strong technical advantage. Imagine that Tor HiddenService’s own traffic is not only fully end-to-end encrypted, but also the host name is the public key, as long as the link is correct, there is no need to worry about man-in-the-middle attacks, and because all traffic passes through the Tor network, you can run any TCP server without a public IP address. At the same time, the attack surface is much less than the WWW service, many users let their home servers refuse all inbound connections, and Tor is a very useful application scenario for remote access and management. In addition, the future development of next generation network tools based on the anonymous network technology foundation represented by Tor is very promising. Ricochet is an absolutely decentralized chat Service based on Tor Hidden Service. Each person’s account is the host name of Onion server on their own machine. All communication is conducted point-to-point directly through Hidden Service. Not only can mass monitors not intercept the content of chats, but they can’t analyze your network of relationships through metadata (experimental project, not recommended for ordinary users). Now public sites such as DuckDuckGo, Blockchain, GnuPG, and even Debian software sources have opened their own Hidden Service, and have been better protected.

Due to space, Alice will not say much here, and will continue to introduce it to you in the future. Thank you for your support. In addition, the developers urge everyone to deploy the Onion protocol as often as possible to promote the technology, and if you’re a website owner, it’s great to have your website support TorHidden Service access.

At the end of the article, Eric Hughes’ 1993 Cypherpunk manifesto is excerpted to commemorate their groundbreaking contributions.

Privacy is a necessity for an open society in the electronic age. “Privacy” is different from “confidentiality” in that “private matters” refer to things that a person does not want the world to know, while “confidential” refers to things that a person does not want anyone to know. Privacy, therefore, refers to the right to selectively disclose information to the public. If the two parties had exchanges, then both have memories, and either can talk about their meetings at will, what’s to stop them? Of course, a law can be enacted to prohibit it, but freedom of speech is also a fundamental condition of an open society, and trumps the right to privacy, so we should not restrict speech. As we ask for privacy, therefore… Both sides just need to know that… Directly relevant information When I buy a magazine with cash at the counter, the cashier doesn’t need to know who I am. When I ask my email provider to send and receive messages, my provider also doesn’t need to know who I am, who I’m talking to, and who’s talking to me; They just need to know how to deliver the information and what fees I should pay. When my identity is revealed by the trading mechanism, I have no privacy. … Open societies also need privacy, and when I speak, I only want my audience to hear those words, and I have no privacy when what I say is available all over the world. … We cannot expect the mercy of governments, corporations, or other faceless giant institutions to grant us privacy. … We must protect our privacy - if we expect to have it at all.

Graphene OS -Hardened security Handbook

Sat, 11 Nov 2023 00:00:00 +0000

GrapheneOS itself is the Hardened system for Android. Recommended by Edward Snowden: I use Graphene OS every day!

The system itself has been enhanced with kernel enhancements, browser enhancements, sandboxed profiles, randomized MAC locations, removal of Google, camera, microphone permissions management, and support for the secure hardware chip HSM technology on newer Pixel phones (Titan M2). But there is still a lot left to tweak to maximize security/privacy.

I’ll show you the method I used here. And how I use the habits of Qubes OS/Tails/Whonix, combined with the system Grapheneos, which is expected and considered to be privacy and anonymity.

Hardened // Security Step 1: Use a different profile for the main system

Immediately after installing the system, a profile must be created for switching. The reason for this is that the main profile is Admin, which is the main directory. The less attacked Admin is, the better. Please use 100% of the other profiles for daily use.

Hardened // Security Step 2: Setup, disable all unneeded features to increase security.

Everything will be done according to the Graphene OS manual.

First we need to turn off the 2G/3G/5G network and use only LTE mode. By default turn off the option to allow sensors. Turn off all USB by default. Turn off Bluetooth, Printing, NFC. Keep Airplane Mode on. Use power saving mode. Screen set password/fingerprint and turn on password randomization. Turn off developer functions Disable OEM mode Turn off positioning, camera, microphone (or cultivate good permissions control, which means you can’t allow permissions on the fly.)

If you’ve done what I said above, you’ve basically got it all set up, and the next Hardened // Security Steps 3 ~ 4 are all about personal taste adjustments, or extreme security.

Hardened // Security Step 3 : Disposable Container Protection Measures

The concept is like a disposable virtual machine, with greater control over permissions / protection against forensics. How to use: In the configuration file, turn on Guest Mode and check the box to remove guest activity. After following the steps above, the disposable container will be completely destroyed whenever you close the guest mode. With GrapheneOS’s perfect Sandbox isolation, this is basically a no-brainer!

Hardened // Security Step 4 : Setting up an environment like Tails / Whonix to connect to the tor network 100% of the time.

Here we use inviZible Pro software to cover the whole network with Tor and exclude the Tor browser from the software. This way the two software will not interfere with each other.

However, it is also important to realize that the security of Tor on Android is very weak anyway. Remember, it is highly recommended to use it with Hardened // Security Step 3!

Hardened // Security Step 5 : Daily Browsing/Anonymous

I’m going to use Whonix’s distinction between pseudonymity and anonymity here. In any case, you should use a profile to separate the two and not merge them together.

For example, for day-to-day browsing, you can only use a VPN on it, or not use a VPN tool. Because this identity means that you (the pseudonym) can know who you are. But not who you really are.

Anonymity. That means you can only use TOR on it, and you can’t use your pseudonym identity, including your account, on Anonymous. (No one knows who you are), which also corresponds to the phrase: what person nothing

Whonix is a compound of two words: who (“what person/s”) and nix (a German word that means “nothing”)

Misperception/actualization clarification:

Everyone on this system thinks that you have to use Tor Browser/Tor related software to maximize the security/anonymity of the system. According to the official GrapheneOS statement, the lack of sandboxing in Tor/Firefox for Android creates an attack surface, including the fact that Firefox’s sandboxing is weak, as described by the developers of the Whonix system.

Also, even using orbot/inviZible Pro to cover the entire system with Tor network is not a good idea, because of the following reasons: Fingerprint / DRM identification, the system mostly relies on profiles for solid security, isolation.

Glossary:

Fingerprint: A fingerprint is the equivalent of a unique identification mechanism for the device. Once a threat/enemy has a fingerprint, they know 100% who that person is and what they are looking for. The theory applies in detail to cell phone software. Browsers. The way to avoid 100% fingerprint recognition in Graphene OS is to use the Tor browser, and all software should run on the Tor browser.

If you do install other software on your system, make sure it is 100% internet free. If it’s a game, or any other software that requires internet access, there is a high chance that your fingerprints will be recognized by the other party.

Announcing Wildland Client v0.1

Fri, 11 Jun 2021 00:00:00 +0000

On behalf of the whole team, I’m proud to announce the first public release of Wildland client! This is a reference implementation of the Wildland protocol, which we have described in our “Why, What and How” (aka W2H) paper.

Leaving aside all the usual disclaimers about how early-beta and for-power-users-only this version really is, I’d like to focus on what’s already working and possible :)

The client functionality

Firstly, you can connect multiple storage backends (S3 buckets, WebDAV servers, Dropbox accounts, etc) and expose them all as one unified file system as shown on the screenshot below. We believe that this is how data management should really be done. Users should not be concerned with which service provider their data is currently being held on. Instead, we should be able to address our data using infrastructure-agnostic addressing. Whether our files live on my Dropbox, local NAS, some p2p-synced storage, or on my local disk, should be of secondary importance, and should not be included in addressing of the data.

Secondly, Wildland natively implements what we call multi-categorization. This means that every data container (the smallest unit of information on Wildland) can be assigned more than one address/path and can be accessed using any of them. Functionality-wise this is similar to having a directory on a traditional filesystem with multiple links, except that on Wildland every path is a first-class citizen. We believe that this feature is vital in making filesystems powerful enough to allow for their use as a primary tool for information and knowledge management. This is a broad and exciting topic and we will be talking more about it in future posts.

Finally, the third highlight of this release is support for what we call cascading addressing. It can be thought of as an “upside-down DNS”, or a bottom-up scalable addressing scheme which does not require any centralized authority, like predefined DNS root servers. It’s important to note that this addressing scheme does not require any blockchain either.

The Wildland demo forests

Aside from today’s release of the client, we’re also opening up for public read-only access two Wildland forests (a forest is a namespace built around one user identity):

The Pandora forest (a reference to the Avatar movie), which is a repository of memos, reports, documents, etc, which we’ve been using internally to help us with Wildland’s development. It’s a bit like a shared Dropbox folder, except that it uses abstracted storage and heavily utilizes multi-categorization.
The Ariadne forest, which is a very small forest containing bridges to other forests, specifically to the Pandora forest, and some other, internal forests, which are encrypted and not exposed for public use.

What’s coming up next?

There are more exciting things ahead of us which we plan to bring to you in the upcoming releases. These can be generally grouped into three areas:

Making Wildland easier to use and set up. This means: better integration on various platforms, easy-to-use GUI, and – last but not least – an Ethereum-based marketplace for storage, followed by user-centered governance and GLM integration, as described in the W2H paper.
Making Wildland faster – primarily optimized mounting of large forests, as well as adding mechanisms for container content caching.
Making Wildland more universal, which includes better integration with more storage backends, integration with various data sources, as well as adding support for compute backends attached to containers, allowing them to actively process the data inside containers.

Our ultimate goal is to provide a set of tools for a user to be able to better manage their digital universe of information. And do so on their own terms, rather than remaining dependent on large service providers to that on behalf of us all.

If you would like to try Wildland today, we suggest you start with this introduction, which is then followed by step-by-step tutorials on how to mount and explore the above-mentioned demo forests, as well as perform many other tasks using Wildland.

The Next Chapter: From the Endpoint to the Cloud

Thu, 25 Oct 2018 00:00:00 +0000

Earlier this year, I decided to take a sabbatical. I wanted to reflect on my infosec work and decide what I would like to focus on in the coming years. As you probably know, I’ve spent the last nine years mostly fighting the battle to secure the endpoint, more specifically creating, developing, architecting, and promoting Qubes OS, as well as the more general concept of “Security through Distrusting”.

Over these past nine years, Qubes OS has grown from a research-inspired proof-of-concept into a reasonably mature, large open-source project with dozens of contributors and tens of thousands of users, including some high-profile security experts.

There are still many challenges ahead for Qubes, however. The primary two are improving hardware compatibility and UX. I think addressing these challenges will mostly be an iterative process of gradual improvement, but it will nonetheless require a lot of time and resources.

Another challenge is the trustworthiness of the x86 platform. Again, I remain optimistic that many things could be improved on this front, as well (see, for example, my proposal for a stateless laptop design). Admittedly, such work will require significantly more resources, but I know there is a lot of interest in this area, and it just needs proper coordination to happen.

Despite all of these interesting challenges (especially with platform trustworthiness), I’ve decided I’d like to switch my focus to something other than endpoint security. While I still believe that the security of our digital lives starts and ends with the trustworthiness of the client devices we use (which today include much more than just laptops and desktops, of course), I also recognize that the state of endpoint device security has significantly improved over the past decade. At the same time, most of our data and activities have migrated from local devices to the cloud.

And yet, there are fundamental problems with cloud trustworthiness, or the lack thereof. These problems could be summarized as the need for users (i.e. all of us) to be forced to trust three different entities: (1) the service providers who own our data (e.g. the vendor of your fitness tracking app), (2) the hosting infrastructure owners, who can both access our data as well as deny us use of the service at their discretion (e.g. AWS, Azure, GCP), and (3) the networking infrastructure operators, who can also selectively cut us off from the services (e.g. to implement some form of censorship).

These are very important problems, in my opinion, and I’d like to work now on making the cloud more trustworthy, specifically by limiting the amount of trust we have to place in it.

So, this month I’ve joined the Golem Project as a Chief Strategy Officer, also doubling as Chief Security Officer (conveniently, both roles have the same CSO acronym :) The double nature of my role emphasizes the close relationships among the long-term roadmap, technical architecture, as well as product- and operations-security in Golem.

Why Golem? Because Golem is a very unique project. Golem has been on a mission to build a “decentralized computer” out of a heterogeneous network of third-party provided computers. Founded two years ago through a very successful crowdfunding campaign, it instantly gained an impressive amount of funding, which allowed it to build a strong development team (incidentally, mostly based in my favorite city – Warsaw).

The distributed funding model has also essentially eliminated two common obstacles most tech startups face: lack of money to hire enough people and the need to implement investors’ (rather than the founders’) agenda. In this respect Golem seems to be as independent as one could realistically imagine.

Most importantly, we (ITL), have already been working with Golem over the past year. During that time I’ve had enough time to get to know some of the key people in the project, understand their personal agendas, and conclude they might be very much inline with my own.

As for the future of Qubes, I don’t think much will change here. In recent years, Marek Marczykowski-Górecki has been effectively leading most of the day-to-day efforts with Qubes OS development, while I’ve focused mostly on long-term architecture planning and various research activities, such as our SGX-related work. Marek will continue to lead Qubes now, so I’m reassured about the future of the project. I will also remain as an advisor to the Qubes OS Project, as well as… its user, though I’ve recently also been embracing other systems, including – of course – the cloud.

Finally, I’d like to thank my (present and past) ITL colleagues. We have undertaken quite a few projects together, of which Qubes OS is probably the most challenging and spectacular. Yet it was hardly the only one. There was also all the (allegedly pioneering) offensive system security research done together with Rafał Wojtczuk and Alex Tereshkin. This research was instrumental in our subsequent work on Qubes OS, and I think it will again prove useful for my work on Golem.

Introducing graphene-ng: running arbitrary payloads in SGX enclaves

Mon, 11 Jun 2018 00:00:00 +0000

A few months ago, during my keynote at Black Hat Europe, I was discussing how we should be limiting the amount of trust when building computer systems. Recently, a new technology from Intel has been gaining popularity among both developers and researchers, a technology which promises a big step towards such trust-minimizing systems. I’m talking about Intel SGX, of course.

Intel SGX caught my attention for the first time about 5 years ago, a little while before Intel has officially added information about it to the official Software Developer’s Manual. I’ve written two posts about my thoughts on this (then-upcoming) technology, which were a superposition of both positive and negative feelings.

Over the last 2 years or so, together with my team at ITL, we’ve been investigating this fascinating technology a bit closer. Today I’d like to share some introductory information on this interesting project we’ve been working on together with our friends at Golem for several months now.

Enclave-based computing

Recently the term “enclave-based” computing has been used increasingly often to describe a form of security container which allows for computations which are protected from the host, such as the external operating system, hypervisor, or even interference from the low-level firmware such as the BIOS/SMM/UEFI.

This is, naturally, in stark contrast to the usual definition of a security container as used today (and as implemented by VMs of various sorts, Linux/Docker containers, etc), which implies protection of the host from whatever code runs inside the container. In other words: the enclaves and the containers represent two different, complementary goals, and it is normally envisioned that both will be deployed at the same time.

It’s not a surprise that people have been thinking about how to protect code and data running on a potentially compromised host for quite some time.

Even I wrote a piece back in 2011, where I discussed how we could attempt to somehow implement trusted execution within an untrusted cloud, utilizing Intel TXT technology (which could be thought of as a logical predecessor of Intel SGX). There has also been very interesting work done by Private Core (now Facebook), starting around 2012 and going on for a few years, unfortunately never released to the public (?). But solutions based on Intel TXT have been clumsy and vulnerable to some fundamental problems, such as e.g. SMM attacks.

Today Intel SGX seems like a very promising candidate (and so far the only one?) offering a reasonably good solution for enclave-based computing.

But Intel is not the only one, there are also other technologies with AMD Secure Encrypted Virtualization (SEV) being another interesting technology. Unfortunately, AMD SEV, while very promising at first sight, does have serious security weaknesses, such as the lack of integrity protection for 2nd-level page tables, and generally is believed to not to offer protection against malicious hypervisors, but rather only against accidental hypervisor vulnerabilities. Hopefully in the future AMD will improve the design and implementation of SEV.

There are also attempts to create so called Trusted Execution Environments (TEEs) on ARM-based systems, typically using the Trust Zone technology, which might resemble the idea of enclave-based computing, in that the payloads are to be protected from the main host OS. These, however, seem quite different to me, architecture-wise, because in case of Trust Zone we really are talking about a classic Security through Isolation Model, just with the hypervisor moved a level down the stack. From the user- and developer- point of view, the similarity might be more striking though. Not being an expert on ARM systems, there is also a possibility I’m missing some crucial innovation here, which makes these solutions also more similar to Intel SGX and AMD SEV.

Intel SGX is a hardware extension, meaning one needs to have a reasonably modern Intel processor in order to be able to use it. Typically this should not be a problem for most modern laptops which are based on most of the recent Intel processors.

But having an SGX-ready hardware is just half of the story, as one also needs specially-designed software to make use of SGX-provided features. And Intel does offer an SDK exactly for this purpose.

So, how does one port an application to be usable within an SGX enclave? As it turns out, that is not an easy task…

Intel SGX challenges and problems

It is not an easy task to port an application to run within an SGX container, because Intel has envisioned SGX as a protection technology for only small parts of the application code and data. Prime example includes protection of only the crypto code (key generation, sign and decrypt operations), while leaving the whole rest of the application wide open to the attacks from the host OS.

This kind of thinking is not entirely new in the industry, where vendors consider the protection of the keys (which usually belong to the vendors, such as e.g. the signing keys for bank transactions or decryption keys for DRM-protected material) to be of primary concern, while putting less emphasis on protection of the actual user data (e.g. the content of the user’s email).

There are more problems with SGX, however.

First, Intel SGX has its own Remote Attestation scheme, which oddly, requires an Intel-operated server, so called Intel Attestation Service (IAS), to be used to verify the proof of attestation. This proof of attestation is called: “quote” and it is the same name as used in “classic” TPM-based Remote Attestation, although the SGX “quote” is a bit different. The Intel’s decision to encrypt it (not just sign), effectively limiting its verification only to Intel-operated servers seems questionable at best. Not only does it introduce a single point of failure to any infrastructure based on SGX, but also gives Intel a possibility to stealthy spoof attestation results for select users, this way facilitating selective plausibly deniable backdooring (note that Intel can always build in a backdoor, just that a generic backdoor within the silicon is hard to be used selectively, and surely is not plausibly deniable).

Another issue with SGX is the requirement for any SGX enclave to be signed with Intel-blessed vendor’s signing key. In other words, it is not possible to load an SGX enclave (other than in debug mode) without entering into a legal contract with Intel. The rationale behind this is believed to be to prevent blackbox malware within enclaves, something I (theoretically) described in the early days of SGX…

(BTW, in the SDM manual, Intel describes a special MSR resister which could be used to provide a hash of a custom, so called, Launch Enclave, seemingly making it possible to get around this requirement to have enclaves signed only by Intel-blessed keys. Unfortunately, it is strongly believed that when one decides to use a custom Launch Enclave, then the Remote Attestation would no longer work, yielding the whole technology mostly useless…)

There are also some implementation-level problems with SGX, which should be mentioned here for completeness. One is the rather small limit for the amount of protected memory accessible for all the enclaves in the system, which is called the Enclave Page Cache, or EPC. On today’s system this amounts to only 128 MB currently. Luckily, the Intel-provided driver (which is part of the SGX SDK) performs on-the-fly paging of enclave memory. This paging goes to the DRAM, not to disk, so while there is some performance penalty imposed by this operation, it is not terrible (in our experiments, for payloads which utilize memory in a more-or-less linear fashion, this amounts to less than 30% overhead). Of course the driver (or the host kernel) only sees the encrypted and integrity-protected pages, so cannot steal, nor meaningfully rearrange them.

Finally, one last problem, which however cannot be dismissed lightly, is the existence of various side-channel attacks, which can leak secrets out of the enclave’s code to the host OS. It is especially challenging to neutralize such side channels, given the fact that the attacker can control the whole host OS. No doubt Intel and other researchers will continue to research this area and likely implement an increasing number of protections against such attacks (most likely on the level of SDK/compiler?).

Running whole applications in SGX enclaves

Despite all the problems with SGX discussed above, I still believe this is a technology with a great potential, and that most of the problems could be solved, or at least improved upon significantly. But to make it truly useful, especially for users, not just vendors, we really need to find generic ways of how to run whole, unmodified applications within SGX enclaves. Pretty much like if they were “VMs” of some kind.

There are two key challenges with achieving this goal, however:

First we need to expose a near-complete API such as POSIX or Win32 (or Linux or Windows syscall API), so that the process running in the enclave could behave as if it ran natively under the host OS. The challenge here is that, naturally, we cannot simply pass through down to the host, because… the host is untrusted in our threat model.
The other challenge is how to meaningfully bind the application inside the enclave to some secret, which could later be used to construct a proof that the application does indeed execute inside a valid SGX enclave. Here we would like to prevent the most obvious attack against enclave-based computing which works by having the attacker… only simulating that the process runs within an enclave, but in reality the execution takes place in normal execution mode (e.g. in case of Intel SGX, the attacker would skip the EINIT/EENTER instructions).

As for solving the first challenge, a few papers and projects have already been presented: Microsoft Haven, SCONE, Graphene SGX, and Panoply.

From the security point of view, the primary difference between these projects, is how they implement the rich OS-like API, which is to be exposed to the application within the enclave.

On the one end of the spectrum are projects which take the “minimal TCB size” approach, which means they try to pass-through as much API call logic to the underlying host OS as possible, while performing some sanity checks on the results returned from the host OS. This approach has been taken by Scone and Panoply, and their rationale is that minimizing the amount of code within an SGX enclave (i.e. the size of TCB), should be the primary goal.

The alternative approach, embraced by Haven and Graphene, has been to “minimize the untrusted interface” to the host OS. The rationale here is that it is the size of the attack surface that matters more, than the amount of code inside the enclave.

I personally agree with the latter approach, because I believe that the size and complexity of the external interface is more critical. After all if there is no way for the attacker to reach the code inside (e.g. pass arbitrary parameters to functions), then it is not so problematic that there might be bugs in this code. By keeping the external interface small and simple, we can assure that with a reasonable degree of assurance in practice. For this reason we have built our solution, discussed below, on top of Graphene.

The second challenge is often neglected, as authors (especially of academic papers) tend to focus more on the first problem, often leaving proof of execution within SGX enclave as “future work” (the MS Haven paper being a notable exception here). Arguably, however, this is the most crucial aspect of any enclave-based solution, as otherwise, no matter how secure the execution environment, the attacker would always be able to essentially ruin the whole scheme by using an emulation attack in which the actual payload will get executed outside of the enclave (or within a real enclave, just such that is controlled by the attacker).

One “little detail” that should also be taken into account is how does the application inside the enclave generate random numbers. Over decades operating systems and applications have come up with various, often surprising ways to generate randomness from different sources of entropy, such as e.g. interrupts, mouse, disk or networking use patterns, etc. When running within an enclave, we should keep in mind that all such “external” events (and only external events can bring entropy, right?) can be fully controlled (at last monitored) by the host OS. This means the host OS can always learn any key the application would generate inside the enclave, which in turn yields any Remote Attestation scheme meaningless. Fortunately, there is a solution to this problem: a hardware instruction for generating randomness: RDRAND. But if the application, for whatever reason, does not use RDRAND (or a proper OS-service, which we intercept in the libOS), but instead tries to employ some Do-It-Yourself method, then we might have a serious problem.

Graphene, graphene-ng and Golem

Together with Golem, we have been working on a solution to the problem of running arbitrary payloads (more specifically: Docker Linux-based containers) for several months now, and I’d like to share a bit of a status of where we are right now, and where we would like to get.

Golem is building a decentralized compute cloud (think EC2 married with SETI@home) and enclave-based computing offers a potential to solve two critical problems any such decentralized network faces: 1) how to ensure users who contribute compute power do indeed execute payloads for which they are being paid, and 2) to offer confidentiality to payloads, so that the owners of the computer performing the computations could not eavesdrop on the data being processed.

We (ITL) on the other hand, are also very interested in enclave-based computing, with one natural application being for use in future versions of Qubes OS.

As mentioned above, we have chosen to base our solution on Graphene SGX, because its architecture seemed the most appealing to us, as explained in the previous section.

Unfortunately, what has looked good on architecture-level (or, to say more directly: on paper), turned out to be much less usable in practice. Any more complex payloads we tried to run under Graphene, such as Blender renderer processing any more complex scenes, revealed dozens of bugs within the Graphene LibOS implementation (mostly race conditions). Luckily these were not security-relevant bugs within our threat model, because in this context we assume the attacks are coming from the host, and not from the payload (and a malicious payload can always be… malicious to itself and its own data, nothing can be done about that).

Golem also considers attacks coming from potentially malicious payloads attacking the host. To prevent them they use a different sandboxing technology, which does not rely on Graphene. This is an important detail to stress, because Graphene actually does advertise itself as a framework to also sandbox the payloads. Our quick analysis of the code revealed, however, that this extra functionality does not work well and could be circumvented relatively easily.

Another problem with the original Graphene project is no meaningful support for proving to the remote party (i.e. the user) that the payload has executed indeed in a proper enclave, properly-protected (the second challenge discussed above).

We’ve spent months finding and fixing the bugs within Graphene and also designing and implementing support for ensuring the remote party the execution and processing of their data has indeed been done within a properly protected enclave.

Our solution for this latter challenge has been designed for batch-processing-like payloads, which take some files on input, perform calculations (e.g. render part of the scene) and then create some output when done. In the future it could be extended to cover more interactive payloads as well. The user is offered a tool, which is responsible for automatically verifying Remote Attestation proof, and encrypting the input file(s) using a public key generated by our framework running within the enclave. This way the user gains assurance that her data could be decrypted only by 1) a payload with specific measurement (hash of the code or hash of the developer’s signing key), 2) running within a proper SGX enclave, 3) on proper, e.g. non-buggy hardware, 4) with the proper set of properties (e.g. not in SGX debug mode).

Given the amount of modifications we have introduced to Graphene, and also because our needs to be able to progress with development significantly faster than the original Graphene project has been doing recently, we’ve decided to fork the project. We will be making the repository public in the coming weeks, most likely after the summer holidays, together with a technical paper describing the challenges I’ve quickly tried to summarize above. For today we just would like to show off a short demo movie below :)

We would also like to thank the original Graphene maintainers and all the contributors for starting this exciting project!

Demo time!

Without further ado, let’s take a look at the demo :)

In the demo above several steps are presented (they are to be done only by the application vendors, not by users of the payloads):

First we’re building the base Docker container which contains the necessary Graphene-based infrastructure: golem-sgx-template
Then we’re creating a Dockerfile for an exemplary application container, in this case for running Blender inside an SGX enclave (golem-sgx-blender):

# Inherit from Golem SGX based template:
FROM golem-sgx-template:latest

# Install Blender:
#  We don't use Ubuntu repo, because it contains some older Blender version.
#  The /wget_and_check.sh is a simple script exposed by the golem-sgx-template Docker
#  which verifies integrity of the downloaded components.
RUN mkdir /blender/                                                                               
RUN /wget_and_check.sh \
    "https://ftp.halifax.rwth-aachen.de/blender/release/Blender2.79/blender-2.79b-linux-glibc219-x86_64.tar.bz2" \
    "/tmp/blender.tar.bz2" \
    "43824a4e0b0c6de6fa34ff224eec44c1cc9f26a95f6f3c8c2558d1c05704183c"
RUN tar xjf /tmp/blender.tar.bz2 --strip-components=1 -C /blender/

# (Automatically) create the manifest for the payload:
WORKDIR /blender/
COPY blender.manifest.template .
RUN /prepare_manifest.sh /blender/blender

# Run the payload under Graphene:
CMD ["/graphene_sgx_run.sh", "blender", \                                                         
     "-t", "1", \                                                                                 
     "-b", "/input/input.blend", \                                                                
     "-P", "/input/change_params.py", \                                                           
     "--", "1920", "1080", "4", "50", "/output/out.png"]           

As we can see the Dockerfile inherits from the golem-sgx-template, and then it’s a pretty standard Dockerfile, except for two things: 1) we need to run a special script to create a so-called manifest for the payload (more on this below), and 2) start the actual payload using the wrapper script (/graphene_sgx_run.sh, also included in the base Docker template), which takes care of running the given executable under Graphene, ensuring specific options are turned on (such as telling it to use SGX-capable PAL).

The payload manifest mentioned above is a crucial thing: it contains a list of all the files which should be made available to the payload running within the SGX enclave. This includes the input and output files, which are automatically encrypted and integrity-protected by a corresponding user-side tool, in such a way that only the proper payload in a valid SGX enclave can read and write them. For Golem applications this process is going to be fully automatic (as seen above), but for other cases the container vendor might need to tweak this manifest by defining different lists of input and output files. The other files, such as the list of all the libraries and other dependencies is generated automatically and this should work for all the cases. More details in the upcoming paper.

Now we’re ready to build and sign the final Docker image. Now the Docker imaging hosting a given payload (in this case a Blender renderer) is finalized (sealed via digital signature made above) and ready for users to run it.

Having such a final docker image, anyone can now run it (e.g. a compute node of the Golem network) with a simple wrapper script, e.g.:

./../docker_run_with_sgx.sh -v "`pwd`/sample_input:/input" -v "`pwd`/render_output:/output" golem-sgx-blender-signed

Future plans and wishes

So, what are our plans and expectations for Intel SGX in the longer term?

First and foremost we look forward to Intel doing the “SGX liberation”, as I like to call it. This liberation should comprise two, independent aspects:

Liberating Remote Attestation, which means Intel should allow anyone, not just its own IAS servers to perform Quote verifications (of course, it’s perfectly fine, and probably justified if Intel will continue to operate and offer IAS to those customers who find such a service convenient),
Allowing to load enclaves signed with any key, not just one “blessed” by Intel. But at the same time still allowing for meaningful Remote Attestation to work. By meaningful attestation I understand one that performs signing of the Quote using keys derived from the processor’s private key, of course.

Other aspects that we believe are crucial in further adoption of SGX technology are:

Allowing for a secure communication path between an enclave and a GPU. This is important for both Golem-like applications, where the payload might want to utilize GPU for computations, as well as for client-like applications, where we would like some user applications, such as e.g. a messaging app or email client, to be able to safely display decrypted content to the user, without the fear of the (untrusted) host OS seeing and capturing it.
Additionally allowing for secure communication with mouse and keyboard (or generally HID devices), to facilitate safe use in client applications, such as just mentioned messaging, email, banking and wallet-like applications.
Increase the amount of EPC to facilitate running big, complex payloads within enclave with minimal performance impact.

Qubes Air: Generalizing the Qubes Architecture

Mon, 22 Jan 2018 00:00:00 +0000

The Qubes OS project has been around for nearly 8 years now, since its original announcement back in April 2010 (and the actual origin date can be traced back to November 11th, 2009, when an initial email introducing this project was sent within ITL internally). Over these years Qubes has achieved reasonable success: according to our estimates, it has nearly 30k regular users. This could even be considered a great success given that 1) it is a new operating system, rather than an application that can be installed in the user’s favorite OS; 2) it has introduced a (radically?) new approach to managing one’s digital life (i.e. an explicit partitioning model into security domains); and last but not least, 3) it has very specific hardware requirements, which is the result of using Xen as the hypervisor and Linux-based Virtual Machines (VMs) for networking and USB qubes. (The term “qube” refers to a compartment – not necessarily a VM – inside a Qubes OS system. We’ll explain this in more detail below.)

For the past several years, we’ve been working hard to bring you Qubes 4.0, which features state-of-the-art technology not seen in previous Qubes versions, notably the next generation Qubes Core Stack and our unique Admin API. We believe this new platform (Qubes 4 represents a major rewrite of the previous Qubes codebase!) paves the way to solving many of the obstacles mentioned above.

The new, flexible architecture of Qubes 4 will also open up new possibilities, and we’ve recently been thinking about how Qubes OS should evolve in the long term. In this article, I discuss this vision, which we call Qubes Air. It should be noted that what I describe in this article has not been implemented yet.

Why?

Before we take a look at the long-term vision, it might be helpful to understand why we would like the Qubes architecture to further evolve. Let us quickly recap some of the most important current weaknesses of Qubes OS (including Qubes 4.0).

Deployment cost (aka “How do I find a Qubes-compatible laptop?”)

Probably the biggest current problem with Qubes OS – a problem that prevents its wider adoption – is the difficulty of finding a compatible laptop on which to install it. Then, the whole process of needing to install a new operating system, rather than just adding a new application, scares many people away. It’s hard to be surprised by that.

This problem of deployment is not limited to Qubes OS, by the way. It’s just that, in the case of Qubes OS, these problems are significantly more pronounced due to the aggressive use of virtualization technology to isolate not just apps, but also devices, as well as incompatibilities between Linux drivers and modern hardware. (While these driver issues are not inherent to the architecture of Qubes OS, they affected us nonetheless, since we use Linux-based VMs to handle devices.)

The hypervisor as a single point of failure

Since the beginning, we’ve relied on virtualization technology to isolate individual qubes from one another. However, this has led to the problem of over-dependence on the hypervisor. In recent years, as more and more top notch researchers have begun scrutinizing Xen, a number of security bugs have been discovered. While many of them did not affect the security of Qubes OS, there were still too many that did. :(

Potential Xen bugs present just one, though arguably the most serious, security problem. Other problems arise from the underlying architecture of the x86 platform, where various inter-VM side- and covert-channels are made possible thanks to the aggressively optimized multi-core CPU architecture, most spectacularly demonstrated by the recently published Meltdown and Spectre attacks. Fundamental problems in other areas of the underlying hardware have also been discovered, such as the Row Hammer Attack.

This leads us to a conclusion that, at least for some applications, we would like to be able to achieve better isolation than currently available hypervisors and commodity hardware can provide.

How?

One possible solution to these problems is actually to “move Qubes to the cloud.” Readers who are allergic to the notion of having their private computations running in the (untrusted) cloud should not give up reading just yet. Rest assured that we will also discuss other solutions not involving the cloud. The beauty of Qubes Air, we believe, lies in the fact that all these solutions are largely isomorphic, from both an architecture and code point of view.

Example: Qubes in the cloud

Let’s start with one critical need that many of our customers have expressed: Can we have “Qubes in the Cloud”?

As I’ve emphasized over the years, the essence of Qubes does not rest in the Xen hypervisor, or even in the simple notion of “isolation,” but rather in the careful decomposition of various workflows, devices, apps across securely compartmentalized containers. Right now, these are mostly desktop workflows, and the compartments just happen to be implemented as Xen VMs, but neither of these aspects is essential to the nature of Qubes. Consequently, we can easily imagine Qubes running on top of VMs that are hosted in some cloud, such as Amazon EC2, Microsoft Azure, Google Compute Engine, or even a decentralized computing network, such as Golem. This is illustrated (in a very simplified way) in the diagram below:

It should be clear that such a setup automatically eliminates the deployment problem discussed above, as the user is no longer expected to perform any installation steps herself. Instead, she can access Qubes-as-a-Service with just a Web browser or a mobile app. This approach may trade security for convenience (if the endpoint device used to access Qubes-as-a-Service is insufficiently protected) or privacy for convenience (if the cloud operator is not trusted). For many use cases, however, the ability to access Qubes from any device and any location makes the trade-off well worth it.

We said above that we can imagine “Qubes running on top of VMs” in some cloud, but what exactly does that mean?

First and foremost, we’d want the Qubes Core Stack connected to that cloud’s management API, so that whenever the user executes, say, qvm-create (or, more generally, issues any Admin API call, in this case admin.vm.Create.*) a new VM gets created and properly connected in the Qubes infrastructure.

This means that most (all?) Qubes Apps (e.g. Split GPG, PDF and image converters, and many more), which are built around qrexec, should Just Work (TM) when run inside a Qubes-as-a-Service setup.

Now, what about the Admin and GUI domains? Where would they go in a Qubes-as-a-Service scenario? This is an important question, and the answer is much less obvious. We’ll return to it below. First, let’s look at a couple more examples that demonstrate how Qubes Air could be implemented.

Example: Hybrid Mode

Some users might decide to run a subset of their qubes (perhaps some personal ones) on their local laptops, while using the cloud only for other, less privacy-sensitive VMs. In addition to privacy, another bonus of running some of the VMs locally would be much lower GUI latency (as we discuss below).

The ability to run some VMs locally and some in the cloud is what I refer to as Hybrid Mode. The beauty of Hybrid Mode is that the user doesn’t even have to be aware (unless specifically interested!) in whether a particular VM is running locally or in the cloud. The Admin API, qrexec services, and even the GUI, should all automatically handle both cases. Here’s an example of a Hybrid Mode configuration:

Another benefit of Hybrid Mode is that it can be used to host VMs across several different cloud providers, not just one. This allows us to solve the problem of over-dependence on a single isolation technology, e.g. on one specific hypervisor. Now, if a fatal security bug is discovered that affects one of the cloud services hosting a group of our VMs, the vulnerability will not automatically affect the security of our other groups of VMs, since the other groups may be hosted on different cloud services, or not in the cloud at all. Crucially, different groups of VMs may be run on different underlying containerization technologies and different hardware, allowing us to diversify our risk exposure against any single class of attack.

Example: Qubes on “air-gapped” devices

This approach even allows us to host each qube (or groups of them) on a physically distinct computer, such as a Raspberry PI or USB Armory. Despite the fact that these are physically separate devices, the Admin API calls, qrexec services, and even GUI virtualization should all work seamlessly across these qubes!

For some users, it may be particularly appealing to host one’s Split GPG backend or password manager on a physically separate qube. Of course, it should also be possible to run normal GUI-based apps, such as office suites, if one wants to dedicate a physically separate qube to work on a sensitive project.

The ability to host qubes on distinct physical devices of radically different kinds opens up numerous possibilities for working around the security problems with hypervisors and processors we face today.

Under the hood: Qubes Zones

We’ve been thinking about what changes to the current Qubes architecture, especially to the Qubes Core Stack, would be necessary to make the scenarios outlined above easy (and elegant) to implement.

There is one important new concept that should make it possible to support all these scenarios with a unified architecture. We’ve named it Qubes Zones.

A Zone is a concept that combines several things together:

An underlying “isolation technology” used to implement qubes, which may or may not be VMs. For example, they could be Raspberry PIs, USB Armory devices, Amazon EC2 VMs, or Docker containers.
The inter-qube communication technology. In the case of qubes implemented as Xen-based VMs (as in existing Qubes OS releases), the Xen-specific shared memory mechanism (so called Grant Tables) is used to implement the communication between qubes. In the case of Raspberry PIs, Ethernet technology would likely be used. In the case of Qubes running in the cloud, some form of cloud-provided networking would provide inter-qube communication. Technically speaking, this is about how Qubes’ vchan would be implemented, as the qrexec layer should remain the same across all possible platforms.
A “local copy” of an Admin qube (previously referred to as the “AdminVM”), used mainly to orchestrate VMs and make policing decisions for all the qubes within the Zone. This Admin qube can be in either “Master” or “Slave” mode, and there can only be one Admin qube running as Master across all the Zones in one Qubes system.
Optionally, a “local copy” of GUI qube (previously referred to as the “GUI domain” or “GUIVM”). As with the Admin qube, the GUI qube runs in either Master or Slave mode. The user is expected to connect (e.g. with the RDP protocol) or log into the GUI qube that runs in Master mode (and only that one), which has the job of combining all the GUI elements exposed via the other GUI qubes (all of which must run in Slave mode).
Some technology to implement storage for the qubes running within the Zone. In the case of Qubes OS running Xen, the local disk is used to store VM images (more specifically, in Qubes 4.0 we use Storage Pools by default). In the case of a Zone composed of a cluster of Raspberry PIs or similar devices, the storage could be a bunch of micro-SD cards (each plugged into one Raspberry PI) or some kind of network storage.

So far, this is nothing radically new compared to what we already have in Qubes OS, especially since we have nearly completed our effort to abstract the Qubes architecture away from Xen-specific details – an effort we code-named Qubes Odyssey.

What is radically different is that we now want to allow more than one Zone to exist in a single Qubes system!

In order to support multiple Zones, we have to provide transparent proxying of qrexec services across Zones, so that a qube need not be aware that another qube from which it requests a service resides in a different zone. This is the main reason we’ve introduce multiple “local” Admin qubes – one for each Zone. Slave Admin qubes are also bridges that allow the Master Admin qube to manage the whole system (e.g. request the creation of new qubes, connect and set up storage for qubes, and set up networking between qubes).

Under the hood: qubes’ interfaces

Within one Zone, there are multiple qubes. Let me stress that the term “qube” is very generic and does not imply any specific technology. It could be a VM under some virtualization system. It could be some kind of a container or a physically separate computing device, such as a Raspberry PI, Arduino board, or similar device.

While a qube can be implemented in many different ways, there are certain features it should have:

A qube should implement a vchan endpoint. The actual technology on top of which this will be implemented – whether some shared memory within a virtualization or containerization system, TCP/IP, or something else – will be specific to the kind of Zone it occupies.
A qube should implement a qrexec endpoint, though this should be very straightforward if a vchan endpoint has already been implemented. This ensures that most (all?) the qrexec services, which are the basis for most of the integration, apps, and services we have created for Qubes, should Just Work(TM).
Optionally, for some qubes, a GUI endpoint should also be implemented (see the discussion below).
In order to be compatible with Qubes networking, a qube should expect one uplink network interface (to be exposed by the management technology specific to that particular Zone), and (optionally) multiple downlink network interfaces (if it is to work as a proxy qube, e.g. VPN or firewalling qube).
Finally, a qube should expect two kinds of volumes to be exposed by the Zone-specific management stack:
- one read-only, which is intended to be used as a root filesystem by the qube (the management stack might also expose an auxiliary volume for implementing copy-on-write illusion for the VM, like the volatile.img we currently expose on Qubes),
- and one read-writable, which is specific to this qube, and which is intended to be used as home directory-like storage. This is, naturally, to allow the implementation of Qubes templates, a mechanism that we believe brings not only a lot of convenience but also some security benefits.

GUI virtualization considerations

Since the very beginning, Qubes was envisioned as a system for desktop computing (as opposed to servers). This implied that GUI virtualization was part of the core Qubes infrastructure.

However, with some of the security-optimized management infrastructure we have recently added to Qubes OS, i.e. Salt stack integration (which significantly shrinks the attack surface on the system TCB compared to more traditional “management” solutions), the Qubes Admin API (which allows for the fine-grained decomposition of management roles), and deeply integrated features such as templates, we think Qubes Air may also be useful in some non-desktop applications, such as the embedded appliance space, and possibly even on the server/services side. In this case, it makes perfect sense to have qubes not implement GUI protocol endpoints.

However, I still think that the primary area where Qubes excels is in securing desktop workflows. For these, we need GUI ~~virtualization~~multiplexing, and the qubes need to implement GUI protocol endpoints. Below, we discuss some of the trade-offs involved here.

The Qubes GUI protocol is optimized for security. This means that the protocol is designed to be extremely simple, allowing only for very simple processing on incoming packets, thus significantly limiting the attack surface on the GUI daemon (which is usually considered trusted). The price we pay for this security is the lack of various optimizations, such as on-the-fly compression, which others protocols, such as VNC and RDP, naturally offer. So far, we’ve been able to get away with these trade-offs, because in current Qubes releases the GUI protocol runs over Xen shared memory. DRAM is very fast (i.e has low latency and super-high bandwidth), and the implementation on Xen smartly makes use of page sharing rather than memory copying, so that it achieves near native speed (of course with the limitation that we don’t expose GPU functionalities to VMs, which might limit the experience in some graphical applications anyway).

However, when qubes run on remote computers (e.g in the cloud) or on physically separate computers (e.g. on a cluster of Raspberry PIs), we face the potential problem of graphics performance. The solution we see is to introduce a local copy of the GUI qube into each zone. Here, we make the assumption that there should be a significantly faster communication channel available between qubes within a Zone than between Zones. For example, inter-VM communication within one data center should be significantly faster than between the user’s laptop and the cloud. The Qubes GUI protocol is then used between qubes and the local GUI qube within a single zone, but a more efficient (and more complex) protocol is used to aggregate the GUI into the Master GUI qube from all the Slave GUI qubes. Thanks to this combined setup, we still get the benefit of a reasonably secure GUI. Untrusted qubes still use the Qubes secure GUI protocol to communicate with the local GUI qube. However, we also benefit from the greater efficiency of remote access-optimized protocols such as RDP and VNC to get the GUI onto the user’s device over the network. (Here, we make the assumption that the Slave GUI qubes are significantly more trustworthy than other non-privileged qubes in the Zone. If that’s not the case, and if we’re also worried about an attacker who has compromised a Slave GUI qube to exploit a potential bug in the VNC or RDP protocol in order to attack the Master GUI qube, we could still resort to the fine-grained Qubes Admin API to limit the potential damage the attacker might inflict.)

Digression on the “cloudification” of apps

It’s hard not to notice how the model of desktop applications has changed over the past decade or so, where many standalone applications that previously ran on desktop computers now run in the cloud and have only their frontends executed in a browser running on the client system. How does the Qubes compartmentalization model, and more importantly Qubes as a desktop OS, deal with this change?

Above, we discussed how it’s possible to move Qubes VMs from the user’s local machine to the cloud (or to physically separate computers) without the user having to notice. I think it will be a great milestone when we finally get there, as it will open up many new applications, as well as remove many obstacles that today prevent the easy deployment of Qubes OS (such as the need to find and maintain dedicated hardware).

However, it’s important to ask ourselves how relevant this model will be in the coming years. Even with our new approach, we’re still talking about classic standalone desktop applications running in qubes, while the rest of the world seems to be moving toward an app-as-a-service model in which everything is hosted in the cloud (e.g. Google Docs and Microsoft Office 365). How relevant is the whole Qubes architecture, even the cloud-based version, in the app-as-a-service model?

I’d like to argue that the Qubes architecture still makes perfect sense in this new model.

First, it’s probably easy to accept that there will always be applications that users, both individual and corporate, will prefer (or be forced) to run locally, or at least on trusted servers. At the same time, it’s very likely that these same users will want to embrace the general, public cloud with its multitude of app-as-a-service options. Not surprisingly, there will be a need for isolating these workloads from interfering with each other.

Some examples of payloads that are better suited as traditional, local applications (and consequently within qubes), are MS Office for sensitive documents, large data-processing applications, and… networking and USB drivers and stacks. The latter things may not be very visible to the user, but we can’t really offload them to the cloud. We have to host them on the local machine, and they present a huge attack surface that jeopardizes the user’s other data and applications.

What about isolating web apps from each other, as well as protecting the host from them? Of course, that’s the primary task of the Web browser. Yet, despite vendors’ best efforts, browser security measures are still being circumvented. Continued expansion of the APIs that modern browsers expose to Web applications, such as WebGL, suggests that this state of affairs may not significantly improve in the foreseeable future.

What makes the Qubes model especially useful, I think, is that it allows us to put the whole browser in a container that is isolated by stronger mechanisms (simply because Qubes does not have to maintain all the interfaces that the browser must) and is managed by Qubes-defined policies. It’s rather natural to imagine, e.g. a Chrome OS-based template for Qubes (perhaps even a unikernel-based one), from which lightweight browser VMs could be created, running either on the user’s local machine, or in the cloud, as described above. Again, there will be pros and cons to both approaches, but Qubes should support both – and mostly seamlessly from the user’s and admin’s points of view (as well the Qubes service developer’s point of view!).

Summary

Qubes Air is the next step on our roadmap to making the concept of “Security through Compartmentalization” applicable to more scenarios. It is also an attempt to address some of the biggest problems and weaknesses plaguing the current implementation of Qubes, specifically the difficulty of deployment and virtualization as a single point of failure. While Qubes-as-a-Service is one natural application that could be built on top of Qubes Air, it is certainly not the only one. We have also discussed running Qubes over clusters of physically isolated devices, as well as various hybrid scenarios. I believe the approach to security that Qubes has been implementing for years will continue to be valid for years to come, even in a world of apps-as-a-service.

Introducing the Next Generation Qubes Core Stack

Tue, 03 Oct 2017 00:00:00 +0000

This is the 2nd post from the “cool things coming in Qubes 4.0” series, and it discusses the next generation Qubes Core Stack version 3, which is the heart of the new Qubes 4.x releases. The previous part discussed the Admin API which we also introduced in Qubes 4.0 and which heavily relies on this new Qubes Core Stack.

Qubes Core Stack vs. Qubes OS

Qubes Core Stack is, as the name implies, the core component of Qubes OS. It’s the glue that connects all the other components together, and which allows users and admins to interact with and configure the system. For the record, the other components of the Qubes system include:

VM-located core agents (implementing e.g. qrexec endpoints used by various Qubes services),
VM-customizations (making the VMs lightweight and working well with seamless GUI virtualization),
Qubes GUI virtualization (the protocol, VM-located agents, and daemons located in the GUI domain which, for now, happens to be the same as dom0),
GUI domain customizations (Desktop Environment customizations, decoration coloring plugin, etc),
The AdminVM distribution (various customizations, special services, such as for receiving and verifying updates, in the future: custom distro),
The Xen hypervisor (with a bunch of customization patches, occasional hardening) or - in the future - some other virtualising or containerizing software or technology,
Multiple “Qubes Apps” (various services built on top of Qubes qrexec infrastructure, such as: trusted PDF and Image converters, Split GPG, safe USB proxies for HID devices, USB proxy for offering USB devices (exposed via qvm-usb), Yubikey support, USB Armory support, etc)
Various ready-to-use templates (e.g. Debian-, Whonix-based), which are used to create actual VMs, i.e. provide the root filesystem to the VMs,
Salt Stack integration

And all these components are “glued together” by the Qubes Core Stack. The diagram below illustrates the location of all these components in the overall system architecture. Unlike many other Qubes architecture diagrams, this one takes an AppVM-centric approach. (Click the image for the full size version.)

There are also a bunch of additional components not shown on the diagram above, and which technically are not part of a Qubes system, but which are instrumental in building of the system:

qubes-builder,
template-builder,
tons of automatic tests,
as well as all the supporting infrastructure for building, testing and distributing Qubes and the updates, and hosting of the website and documentation.

As you can see Qubes is a pretty complex creature, and the Qubes Core Stack is central to its existence.

Qubes VMs: the building blocks for Explicit Partitioning Model

Qubes implements explicit partitioning security model, which means that users (and/or admins) can define multiple security domains and decide what these domains can and cannot do. This is a different model than the popular sandboxing model, as implemented by increasingly many applications and products today, where every application is automatically sandboxed and some more-or-less pre-defined set of rules is used to prevent behaviour considered “unsafe” (whatever that might mean…). I believe the explicit partitioning model provides many benefits over the sandboxing model, among the most important one being that it is information-oriented, rather than application-oriented. In other words it tries to limit damage to the (user’s) data, rather to the (vendor’s) code and infrastructure.

There have always been a few different kinds of VMs in Qubes: AppVMs, Template VMs, Standalone VMs, NetVMs, ProxyVMs, DispVM, etc. In Qubes 4 we have slightly simplified and cleaned up these categories.

First we’ve hidden the PV vs HVM distinction. Now each VM has a property named virt_mode which is used to decide whether it should be virtualized using Xen para-virtualization (PV), full virtualization with auxiliary qemu in isolated “stub domain” (HVM), or – in the near future – as full virtualization without qemu and the additional stub domain (PVH). This means we no longer classify VMs as PV vs HVMs, because every VM can be easily switched between various modes of virtualization with a flip of a property.

We also no longer distinguish between AppVMs, NetVMs and Proxy VMs. Instead, each VM has a property provides_network, which is false by default, except for the VMs which we want to expose networking to other VMs (e.g. because they might have some networking device assigned, such as cellular modem or WiFi device, or because they act as VPNs or proxies of some sort).

We discuss what the properties are and how to check/set them, later in this article.

So, to recap, starting with Qubes 4.0 we have only the following classes of VMs:

AppVM - covers what we called AppVMs, NetVMs, and ProxyVMs in earlier Qubes releases,
DispVM - defined by not having a persistent private image across VM reboots (see also below),
TemplateVM - these provide root filesystem for AppVMs, the semantics haven’t changed in Qubes 4,
StandaloneVM - these are like AppVMs, but are not based on any TemplateVM,
AdminVM a singleton (i.e. a class which has only one instance), which represents the Administrator VM (Up until recently called dom0, a term we’re departing from, given it is Xen-specific).

One can list all the VM classes known to the stack using the following command:

[user@dom0 ~]$ qvm-create --help-classes

BTW, we’ve been recently promoting the use of alternative names to “VM(s)”, such as domain(s), and - more recently - “qube(s)”. This is to stress the connection between Qubes and virtualization technology is less tight that many might perceive. Another reason is that, we believe, for many users these alternative terms might be more friendly than “VMs”, which apparently have strong technical connotation. The author is quite used to the term VM, however, so should be excused for (ab)using this term throughout her writings.

Properties, Features and Tags

Each VM (domain) in Qubes can have a number of properties, features and tags, which describe both how it should behave, as well as what features or services it offers to other VMs:

Properties, as the name suggests, are used to change the behaviour of the VM and/or how it is treated by the Qubes Core Stack. The virt_mode property mentioned above is a good example. For a list of other properties, take a look here. A command which can be used to list, read and set properties for a VM is qvm-prefs (see below).
Features are very similar to properties, except that they are mostly opaque to the Core (unlike properties, which are well defined and sanitized). Features are essentially a dictionary of ‘key=value’ pairs assigned to each VM. They can be used in various places outside of the main logic of the Core Stack, specifically in Core Extensions. A new tool in Qubes 4 used to inspect or set features is called qvm-features.
A good example of a mechanism implemented on top of features are services, which have been reimplemented in Qubes 4. Services are used to let the VM agents know about whether various additional services, such as Network Manager, should be enabled or not. Indeed, the qvm-service tool now (i.e. in Qubes 4.0) internally simply creates features named service.XYZ, where XYZ is the name of the service passed to qvm-service. It also takes care of interpreting the true/false values.
Finally, each VM can also have some tags associated with it. Unlike features, tags do not have a value – a VM can either have a specific tag (“can be tagged with it”) or not. Unlike properties, they are not interpreted by any core logic of the Core Stack. The sole purpose of tags is that they can be used for qrexec policy rules, as discussed below. (This is also the reason why we wanted to keep them absolutely simple – any complexity within qrexec policy parsing code would definitely be asking for troubles…).

Last but not least, we should mention that each of the VM has a unique name associated with it. VM names in Qubes are like filenames in the filesystem – not only they are unique, but they also are used as a primary way of identifying VMs, especially for security-related decisions (e.g. for qrexec policy construction, see below).

And just as one can get away with using generic tagging schemes in place of referring to paths and filenames in some security systems, similarly in Qubes OS one can refer to tags (mentioned above and discussed later) in the policy (but currently not in firewalling rules).

Internally, a VM’s name is implemented as the property name and thus can be read using qvm-prefs. It cannot be changed, however, because, starting with Qubes 4.0, we treat it as an immutable property. User-exposed tools, however, do have a “VM rename” operation, which is implemented as creating a copy of the VM with the new name and removing the old VM. Thanks to the new volume manager we also introduced in Qubes 4 (and which will be the topic of another post), this operation is actually very cheap, disk-wise.

So, let us now start a console in AdminVM (dom0) and play a bit with these mechanisms.

Let’s start with something very simple and let us take a look at the properties of the AdminVM (in the current Qubes implemented by Xen’s dom0):

[user@dom0 ~]$ qvm-prefs dom0
default_dispvm  D  fedora-25-dvm
label           -  black
name            D  dom0
qid             D  0
uuid            D  00000000-0000-0000-0000-000000000000

As we can see, there aren’t many properties for the AdminVM, and none of them can be modified by the user or admin. While we’re here, we should mention there exists a similar tool, qubes-prefs which allows one to view and modify the global system properties, and these properties should not be confused with the properties of the AdminVM:

[user@dom0 ~]$ qubes-prefs
check_updates_vm       D  True
clockvm                -  sys-net
default_dispvm         -  fedora-25-dvm
default_fw_netvm       D  None
default_kernel         -  4.9.45-21
default_netvm          -  sys-firewall
default_pool           D  lvm
default_pool_kernel    -  linux-kernel
default_pool_private   D  lvm
default_pool_root      D  lvm
default_pool_volatile  D  lvm
default_template       -  fedora-25
stats_interval         D  3
updatevm               -  sys-firewall

Now, let’s inspect an ordinary AppVM, say work (which is one of the default VMs created by the installer, but, of course, the user might have it named differently):

[user@dom0 ~]$ qvm-prefs work
autostart             D  False
backup_timestamp      D
debug                 D  False
default_dispvm        D  fedora-25-dvm
default_user          D  user
gateway               D
include_in_backups    D  True
installed_by_rpm      D  False
ip                    D
kernel                D  4.9.45-21
kernelopts            D  nopat
label                 -  blue
mac                   D  00:16:3E:5E:6C:00
maxmem                D  4000
memory                D  400
name                  -  work
netvm                 -  None
provides_network      D  False
qid                   -  8
qrexec_timeout        D  60
stubdom_mem           U
stubdom_xid           D  6
template              -  fedora-25
template_for_dispvms  D  False
updateable            D  False
uuid                  -  fab9a577-2531-4971-bce1-ca0c9b511f27
vcpus                 D  2
virt_mode             D  hvm
visible_gateway       D
visible_ip            D
visible_netmask       D
xid                   D  5

We see lots of properties which have a D flag displayed, which indicates the property uses the core-provided default value. In most cases the user is able to override these values for specific VMs. For example the virt_mode property is by default set to hvm for all AppVMs, which means that full virtualization mode is used for the domain, but we can override this for specific VM(s) with pv and thus force them to be virtualized using para-virtualization mode, which is what the installer scripts do for the sys-net and sys-usb VMs in Qubes 4.0-rc1 in order to work around problems with PCI passthrough support that we’ve observed on some platforms.

As a simple exercise with changing the property, we can switch off networking for one of the AppVMs by setting its netvm property to an empty value:

[user@dom0 ~]$ qvm-prefs --set work netvm ""

We can now confirm that indeed the VM named work has no network. E.g. we can use the qvm-ls command (it should print - in the column which indicated the netvm), or open a terminal in the work VM itself and try to see if there is a virtual networking device exposing the network (we can e.g. use the ifconfig or ip a commands).

Instead of setting an empty value as the netvm, we could have also set some other VM, thus forcing the networking traffic to pass through that other VM, e.g. to route all the VM’s traffic through a Whonix Tor gateway:

[user@dom0 ~]$ qvm-prefs --set work netvm sys-whonix

The VM which we want to use as the provider of networking services to other VMs, such as sys-whonix in the example above, should have its provides_network property set to true.

To revert back to the system-default netvm (as specified by qubes-prefs), we can use the --default switch:

[user@dom0 ~]$ qvm-prefs --default work netvm

Now let us take a look at the features and services. It might be most illustrative to look at both of these mechanisms together. Let’s suppose we would like to enable the Network Manager service in some VM. Perhaps we would like to use it to create a VPN connection terminated in one of the AppVMs, say in the work VM. (An alternative option would be to create a separate VM, say work-vpn, set its provides_network property to true, enable the network-manager service in there, and use it to provide networking to the other VMs, e.g. work.)

To enable the Network Manager service and a handy widget that shows the status of the VPN, we can do:

[user@dom0 ~]$ qvm-service --enable work network-manager

This should result in the following output:

[user@dom0 ~]$ qvm-service work
network-manager  on

We see the service is enabled, and if we restart the work VM, we should see the Network Manager widget appear in the tray.

Now let’s view the features of this same VM:

[user@dom0 ~]$ qvm-features work
service.network-manager  1

We can now clearly see how the services are internally implemented on top of features. This is done internally by the qvm-services command and by the scripts in the VMs, which interpret specifically-formatted features as services. The qvm-service tool also takes care about properly setting and interpreting the true/false and empty/non-empty strings to make sense for services states (on/off), eliminating user mistakes.

One thing left for us to explore are tags, discussed above, but we will defer this discussion until a later chapter, which talks about qrexec policy, as tags are exclusively for use within policies. For now, suffice to say, that there is a dedicated tool, qvm-tags used to check and set/modify the tags for each of the VM (in addition to some tags being automatically created by the Core Stack, such as e.g. the created-by-XYZ tag, discussed in the previous article on Admin API).

As a final note we should stress one more time that VM names, despite being normal property, are special in that the are immutable. In order to change the name one needs to 1) create a new VM, 2) import the volume from the original VM (using admin.vm.volume.CloneFrom and admin.vm.volume.CloneTo calls), as well as 3) copy all the properties, tags, and features from the original VM, and 4) remove the original VM.

Each of these operations can be controlled independently via qrexec policy. While this might seem like a superficial complication at first sight, we believe it allows to simplify the implementation, as well as to minimize the amount of potential mistakes when creating policies, notably when there is more than one management VMs in the system, such as e.g. the (de-privileged) GUI domain and some corporate-owned (but also semi-privileged) management VM. This has been discussed in more details in the previous post on Admin API.

Disposable VMs redesigned

We have also redesigned how Disposable VMs (DispVMs) work in Qubes 4.0. Before 4.0, DispVMs had two kinds of largely unrelated characteristics: 1) being disposable, i.e. having no persistent private image, and 2) booting from pre-created savefiles in order to save on startup time.

In Qubes 4.0 we have redefined DispVMs solely by this first property, i.e. private-image-not-having, which we believe to be the essential characteristics of a Disposable VM. The underlying mechanism, e.g. whether the VM is restored from a savefile to speed up the startup time, might or might not be implemented for any VM and should not concern the user.

Another major change to DispVMs semantics in Qubes 4 (largely allowed by this relaxation of DispVM definition) is that any of the AppVMs can be now used as a template for the DispVM.

This provides lots of flexibility, e.g. it’s easy to create a customized DispVM for signing PDF documents, or various disposable service VMs, such as Net- and VPN- VMs. One just creates a normal AppVM, sets everything up there, e.g. loads keys, configures VPNs, connects to known WiFi networks, and then shuts down the AppVM and in place of it starts a DispVM based on that AppVM.

This way, whenever one has a gut feeling that something’s wrong with the VM (e.g. because one just connected a USB stick to copy some slides), it’s easy to just restart the DispVM in question and benefit from a new clean root filesystem! Admittedly sometimes this might not be enough to get rid of an infection, as discussed in the recent post on Compromise Recovery, but for many cases this feature is highly desirable.

But opening up this flexibility comes at a price. We have to be careful about which VMs can create which DispVMs. After all, it would be a disaster to allow your casual Internet-browsing AppVM to spawn a DispVM based on your sensitive work AppVM. The casual Internet-browsing VM could have the new DispVM open a malicious file that compromises the DispVM. The compromised DispVM would then be able to leak sensitive work-related data, since it uses the work VM as its template.

There are two mechanisms in place to prevent such mistakes:

Each AppVM has a property called template_for_dispvms, which controls whether this VM can serve as a template for Disposable VMs (i.e., whether any DispVMs based on this VM are allowed in the system. By default, this property is false for all AppVMs and needs to be manually enabled.
The choice of the template (i.e. specific AppVM) for the Disposable VM must be provided by the qrexec policy (which the source VM cannot modify), and defaults to the source VM’s default_dispvm property, which by default has a value as specified via qubes-prefs. The resulting AppVM must have the template_for_dispvms property set, or otherwise an error will occur.

Here we will take a look at how the template can be specified explicitly when starting the DispVM from dom0:

[user@dom0 ~]$ qvm-run --dispvm=work --service qubes.StartApp+firefox
Running 'qubes.StartApp+firefox' on $dispvm:work
$dispvm:work: Refusing to create DispVM out of this AppVM, because template_for_dispvms=False

As mentioned above, we also need to explicitly enable use of the work AppVM as Disposable VMs templates:

[user@dom0 ~]$ qvm-prefs --set work template_for_dispvms True
[user@dom0 ~]$ qvm-run --dispvm=work --service qubes.StartApp+firefox
Running 'qubes.StartApp+firefox' on $dispvm:work

We will look into how the DispVM can be specified via qrexec policy in a later chapter below.

Qubes Remote Execution (qrexec): the underlying integration framework

Qubes OS is more than just a collection of isolated domains (currently implemented as Xen VMs). The essential feature of Qubes OS, which sets it apart from ordinary virtualization systems, is the unique way in which it securely integrates these isolated domains for use in a single endpoint system.

It’s probably accurate to say that the great majority of our efforts is on building this integration in a manner so that it doesn’t ruin the isolation that Xen provides to us.

There are several layers of integration infrastructure in Qubes OS, as depicted in the following diagram (click for full size) and described below:

First, there is the Xen-provided mechanism based on shared memory for inter-VM communication (it’s called “grant tables” in Xen parlance). This is then used to implement various Xen-provided drivers for networking and virtual disk (called “block backends/frontends” by Xen). Generally, any virtualization or containerization system provides some sort of similar mechanisms, and we could easily use of any them on Qubes, because our Core Stack uses libvirt to configure these mechanisms. Qubes does have some specific requirements, however, the most unique one being that we want to put the backends in unprivileged VMs, rather than in dom0 (or host or root partition, however it is called in other systems). This is one of the reasons why Xen still is our hypervisor of choice – most (all?) other systems assume all the backends to be placed in the privileged domain, which clearly is undesirable from the security point of view.
Next, we have the Qubes infrastructure layer, which builds on top of the Xen-provided shared memory communication mechanism. The actual layer of abstraction is called “vchan”. If we moved to another hypervisor, we would need to have vchan implemented on top of whatever inter-VM infrastructure that other hypervisor was to make available to us. Various Qubes-specific mechanisms, such as our security-optimized GUI virtualization, builds on top of vchan.
Finally, and most importantly, there is the Qubes-specific qrexec infrastructure, which also builds on top of vchan, and which exposes socket-like interfaces to processes running inside the VMs. This infrastructure is governed by a centralized policy (enforced by the AdminVM). Most of the Qubes-specific services and apps are built on top of qrexec, with the notable exception of GUI virtualization, as mentioned earlier.

It’s important to understand several key properties of this qrexec infrastructure, which distinguish it from other, seemingly similar, solutions. First, qrexec does not attempt to perform any serialization (à la RPC). Instead, it exposes only a simple “pipe”, pretty much like a TCP layer. This allows us to keep the code which handles the incoming (untrusted) data very simple. Any kind of (de-)serialization and parsing is offloaded to the specific code which has been registered for that particular service (e.g. qubes.FileCopy).

This might seem like a superficial win. After all, the data needs to be de-serialized and parsed somewhere, so why would it matter where exactly? True, but what this model allows us to do is to selectively decide how much risk we want to take for any specific domain by allowing (or not) specific service calls from (specific or all) other domains.

For example, by decoupling the (more complex) logic of data parsing as used by the qubes.FileCopy service from the core code of qrexec we can eliminate potential attacks on the qubes.FileCopy server code by not allowing e.g. any of the VMs tagged personal to issue this call to any of the VMs tagged work, while at the same time still allowing all of these VMs to communicate with the default clockvm (by default sys-net, adjustable via policy redirect as discussed below) to request the qubes.GetDate service. We would not have such a flexibility in trust partitioning if our qrexec infrastructure had serialization built in (e.g. if it was implementing a protocol like DBus between VMs). Of course, specific services might decide to use some complex serializing protocols on top of qrexec (e.g. DBus) very easily, because qrexec connection is seen as a socket by applications running in the VMs.

Another important characteristic is the policing of qrexec, which is something we discuss in the next section.

Let’s write a simple qrexec service, which would be illustrative for what we just discussed. Imagine we want to get the price of a Bitcoin (BTC), but the VM where we need it has no network connectivity, perhaps because it contains some very sensitive data and we want to cut off as much interfaces to the untrusted external world as possible.

In the untrusted VM we will create our simple server with the following body:

curl -s https://blockchain.info/q/24hrprice

The above should be pasted into the /usr/local/etc/qubes-rpc/my.GetBTCprice file in our untrusted AppVM (let’s name it… untrusted). Let’s also test if the service work (still from the untrusted VM):

[user@untrusted ~]$ sudo chmod +x /etc/qubes-rpc/my.GetBTCprice
[user@untrusted ~]$ /usr/local/etc/qubes-rpc/my.GetBTCprice
(...)

We should see the recent price of Bitcoin displayed.

Now, let’s create a network-disconnect, trusted AppVM called wallet:

[user@dom0 ~]$ qvm-create wallet -l blue --property netvm=""
[user@dom0 ~]$ qvm-ls

And now from a console in this new AppVM, let’s try to call our newly created service:

[user@wallet ~]$ qrexec-client-vm untrusted my.GetBTCprice
(...)

The above command will invoke the (trusted) dialog box asking to confirm the request and select the destination VM. For this experiment just select untrusted from the “Target” drop-down list and acknowledge (in case you wonder why the target VM is specified twice – once in the requesting VM and for the 2nd time in the trusted dialog box: we will discuss it in the next section). You should see the price of the bitcoin printed as a result.

Before we move on to discussing the flexibility of qrexec policies, let’s pause for a moment and recap what has just happened:

The network-disconnected, trusted VM called wallet requested the service my.GetBTCprice from a VM named untrusted. The wallet VM had no way to get the price of BTC, because it has no networking (for security).
After the user confirmed the call (by clicking on the trusted dialog box, or by having a specific allow policy, as discussed below), it got passed to the destination VM: untrusted.
The qrexec agent running in the untrusted VM invoked the handling code for the my.GetBTCprice service. This code, in turn, performed a number of complex actions: it TCP-connected to some server on the internet (blockchain.info), performed some very complex crypto handshake to establish an HTTPS connection to that server, then retrieved some complex data over that connection, and finally returned the price of Bitcoin. There’s likely hundreds of thousands lines of code involved in this operation.
Finally, whatever the my.GetBTCprice service returned on its stdout was automagically taken by qrexec agent and piped back to the requesting VM, our wallet VM.

The wallet VM got the data it wanted without the need to get involved in all these complex operations, which take hundreds of thousands of lines of code talking to untrusted computers over the network. That’s how we can improve the security of this process without spending effort on auditing or hardening the programs used (e.g. curl).

Of course that was a very simple example. But a very similar approach is used among many Qubes services. E.g. system updates for dom0 are downloaded in untrusted VMs and exposed to (otherwise network-disconnected) dom0 via the qubes.ReceiveUpdates service (which later verifies digital signatures on the packages). Another example is qubes.PdfConvert, which offloads the complex parsing and rendering of PDFs to Disposable VMs and retrieves back only a very simple format that is easily verified to be non-malicious. This simple format is then converted back to a (now trusted) PDF.

More expressive qrexec policies

Because pretty much everything in Qubes which provides integration over the compartmentalized domains is based on qrexec, it is imperative to have a convenient (i.e. simple to use), secure (i.e. simple in implementation) yet expressive enough mechanism to control who can request which qrexec services from whom. Since the original qrexec policing was introduced in Qubes release 1, the mechanism has undergone some slight gradual improvements.

We still keep the policy as a collection of simple text files, located in /etc/qubes-rpc/policy/ directory in the AdminVM (dom0). This allows for automating of the policy packaging into RPM (trusted) packages, as well policy customization from within our integrated Salt Stack via (trusted) Salt state files.

Now, one of the coolest features we’ve introduced in Qubes 4.0 is the ability to tag VMs and use these to make policy decisions.

Imagine we have several work-related domains. We can now tag all them with some tag of our choosing, say work:

[user@dom0 user ~]$ qvm-tags itl-email add work
[user@dom0 user ~]$ qvm-tags accounting add work
[user@dom0 user ~]$ qvm-tags project-liberation add work

Now we can easily construct a qrexec policy, e.g. to constrain the file (or clipbooard) copy operation, so that it’s allowed only between the VMs tagged as work (while preventing file transfer to and from any VM not tagged with work) – all we need is to add the following 3 extra rules in the policy file for qubes.FileCopy:

[user@dom0 user ~]$ cat /etc/qubes-rpc/polisy/qubes.FileCopy
(...)
$tag:work   $tag:work   allow
$tag:work   $anyvm      deny
$anyvm      $tag:work   deny

$anyvm      $anyvm      ask

We can do the same for clipboard, just need to place the same 3 rules in qubes.ClipboardPaste policy file.

Also, as already discussed in the previous post on Admin API, the Core Stack automatically tags VMs with created-by-XYZ tag, where XYZ is replaced by the name of the VM which invoked the admin.vm.Create* service. This allows to automatically constrain power of specific management VMs to only manage its “own” VMs, and not others. Please refer to the article linked above for the examples.

Furthermore, Disposable VMs can also be referred to via tags in the policy, for example:

# Allow personal VMs to start DispVMs created by the user via guidom:
$tag:created-by-guidom $dispvm:$tag:created-by-guidom allow

# Allow corpo-managed VMs to start DispVMs based on corpo-owned AppVMs:
$tag:created-by-mgmt-corpo $dispvm:$tag:created-by-mgmt-corpo allow

Of course, we can also explicitly specify Disposable VMs using the $dispvm:<name_of_appvm_to_use_as_template> syntax, e.g. to allow AppVMs tagged work to request qrexec service from a Disposable VM created off work-printing AppVM (as noted earlier, the “work-printing” would need to have it’s property template_for_dispvms set for this to work):

$anyvm    $dispvm:work-printing allow

In Qubes 4.0 we have also implemented more strict control over the destination argument for qrexec calls. Until Qubes 4.0, the source VM (i.e., the VM that calls a service) was responsible for providing a valid destination VM name to which it wanted to direct the service call (e.g. qubes.FileCopy or qubes.OpenInVM). Of course, the policy always had the last say in this process, and if the policy had a deny rule for the specific case, the service call was dropped.

What has changed in Qubes 4.0 is that whenever the policy says ask (in contrast to allow or deny), then the VM-provided destination is essentially ignored, and instead a trusted prompt is displayed in dom0 to ask the user to select the destination (with a convenient drop down list).

(One could argue that the VM-provided destination argument in qrexec policy call is not entirely ignored, as it might be used to select a specific qrexec policy line, in case there were a few matching, such as:

$anyvm work allow
$anyvm work-web ask,default_target=work-web
$anyvm work-email ask

In that case, however, the VM-provided call would still be overridden by the user choice in case the rule #2 was selected.)

A prime example of where this is used is the qubes.FileCopy service. However, we should note that for most other services, in a well configured system, there should be very few ask rules. Instead most policies should be either allow or deny, thereby relieving the user from having to make a security decision with every service invocation. Even the qubes.FileCopy service should be additionally guarded by deny rules (e.g. forbidding any file transfers between personal and work-related VMs), and we believe our integrated Salt Management Stack should be helpful in creating such policies in larger deployments of Qubes within corporations.

Here comes in handy another powerful feature of qrexec policy: the target= specifier, which can be added after the action keyword. This forces the call to be directed to the specific destination VM, no matter what the source VM specified. A good place to make use of this is in policies for starting various Disposable VMs. For example, we might have a special rule for Disposable VMs which can be invoked only from VMs tagged with work (e.g. for the qubes.OpenInVM service):

$tag:work $dispvm allow,target=$dispvm:work-printing
$anyvm    $dispvm:work-printing deny 

The first line means that every DispVM created by a VM tagged with work will be based on (i.e., use as its template) the work-printing VM. (Recall that, in order for this to succeed, the work-printing VM also has to have its template_for_dispvms property set to true.) The second line means that any other VM (i.e., any VM not tagged with work) will be denied from creating DispVMs based on the work-printing VM.

`qubes.xml`, `qubesd`, and the Admin API

Since the beginning, Qubes Core Stack kept all the information about the Qubes system’s (persistent) configuration within the /var/lib/qubes/qubes.xml file. This has included information such as what VMs are defined, on which templates they are based, how they are network-connected, etc. This file has never been intended to be edited by the user by hand (except for rare system troubleshooting). Instead, Qubes has provided lots of tools, such as qvm-create, qvm-prefs and qubes-prefs, and many more, which operate or make use of the information from this file.

In Qubes 4.x we’ve introduced the qubesd daemon (service) which is now the only entity which has direct access to the qubes.xml file, and which exposes a well-defined API to other tools. This API is used by a few internal tools running in dom0, such as some power management scripts, qrexec policy checker, and qubesd-query(-fast) wrapper, which in turn is used to expose most parts of this API to other VMs via the qrexec infrastructure. This is what we call Admin API, and which we I described in the previous post. While the mapping between Admin API and the internal qubesd API is nearly “one-to-one”, the primary difference is that Admin API is subject to policy mechanism via qrexec, while the qubesd-exposed API is not policed, because it is only exposed locally within the dom0. This architecture is depicted below.

As discussed in the post about Admin API, we have put lots of thought into designing the API in such a way as to allow effective split between the user and admin roles. Security-wise this means that admins should be able to manage configurations and policies in the system, but not be able to access the user data (i.e. AppVMs’ private images). Likewise it should be possible to prevent users from changing the policies of the system, while allowing them to use their data (but perhaps not export/leak them easily outside the system).

For completeness I’d like to mention that both qrexec and firewalling policies are not included in the central qubes.xml file, but rather in separate locations, i.e. in /etc/qubes-rpc/policy/ and /var/lib/qubes/<vmname>/firewall.xml respectively. This allows for easy updating of the policy files, e.g. from within trusted RPMs that are installed in dom0 and which might be brining new qrexec services, or from whatever tool used to create/manage firewalling policies.

Finally, a note that qubesd (as in “qubes-daemon”) should not be confused with [qubes-db][https://github.com/QubesOS/qubes-core-qubesdb] (as in “qubes-database”). The latter is a Qubes-provided, security-optimized abstraction for exposing static informations from one VMs to others (mostly from AdminVM), and which is used e.g. for the agents in the VMs to get to know the VM’s name, type and other configuration options.

The new attack surface?

With all these changes to the Qubes Core Stack, an important question comes to mind: how do all these changes affect the security of the system?

In an attempt to provide somewhat meaningful answer to that question, we should first observe that there exists a number of obvious configurations (including the default one) of the system, in which there should be no security regression compared to previous Qubes versions.

Indeed, by not allowing any other VM to access the Admin API (which is what the default qrexec policy for Admin API does), we essentially reduce the attack surface onto the Core Stack to what is has been in the previous versions (modulo potentially more complex policy parser, as discussed below).

Let us now imagine exposing some subset of the Admin API to select, trusted management VMs, such as the upcoming GUI domain (in Qubes 4.1). As long as we consider these select VMs as “trusted”, again the situation does not seem to be any worse that what it was before (we can simply think of dom0 as having comprised these additional VMs in previous versions of Qubes. Certainly there is no security benefit here, but likewise there is no added risk).

Now let’s move a step further and relax our trustworthiness requirement for this, say, GUI domain. We will now consider it only “somewhat trustworthy”. The whole promise of the new Admin API is that, with a reasonably policed Admin API (see also the previous post), even if this domain gets compromised, this will not result in full system compromise, and ideally only in some kind of a DoS where none of the user data will get compromised. Of course, in such a situation there is additional attack surface that should be taken into account, such as the qubesd-exposed interface. In case of a hypothetical bug in the implementation of the qubesd-exposed interface (which is heavily sanitized and also written in Python, but still) the attacker who compromised our “somewhat trustworthy” GUI domain might compromise the whole system. But then again, let’s remember that without the Admin API we would not have a “somewhat trustworthy” GUI domain in the first place, and if we assume it was possible for the attacker to compromise this VM, then she would also be able to compromise dom0 in earlier Qubes versions. That would be fatal without any additional preconditions (e.g. for a bug in qubesd).

Finally, we have the case of a “largely untrusted” management VM. The typical scenario could be a management VM “owned” by an organization/corporation. As explained in the previous post, the Admin API should allow us to grant such a VM authority over only a subset of VMs, specifically only those which it created, and not any other (through the convenient created-by-XYZ tags in the policy). Now, if we consider this VM to become compromised, e.g. as a result of the organization’s proprietary management agents getting compromised somehow, then it becomes a very urging question to answer how buggy the qubesd-exposed interface might be. Again, on most (all?) other client system, such a situation would be fatal immediately (i.e. no additional attacks would be required after the attacker compromised the agent), while on Qubes this would only be the prelude for trying another attacks to get to dom0.

One other aspect which might not be immediately clear is the trade-off between a more flexible architecture, which e.g. allows to create mutually untrusted management VMs on the one hand, and the increased complexity of e.g. the policy checker, which is required to now also understand new keywords such as the previously introduced $dispvm:xyz or $tag:xyz. In general we believe that if we can introduce a significant new security improvement on architecture level, which allows to further decompose the TCB of the system, than it is worth it. This is because architecture-level security should always go first, before implementation-level security. Indeed, the latter can always be patched, and in many cases won’t be critical (because e.g. smart architecture will keep it outside of the TCB), while the architecture very often cannot be so easily “fixed”. In fact this is the prime reason why we have Qubes OS, i.e. because fixing of the monolithic architecture of the mainstream OSes has seemed hopeless to us.

Summary

The new Qubes Core Stack provides a very flexible framework for managing a compartmentalized desktop (client) oriented system. Compared to previous Qubes Core Stacks, it offers much more flexibility, which translates to ability to further decompose the system into more (largely) mutually untrusting parts.

Some readers might wonder how does the Qubes Core Stack actually compare to various popular cloud/server/virtualization management APIs, such as OpenStack/EC2 or even Docker?

While at first sight there might be quite a few similarities related to management of VMs or containers, the primary differentiating factor is that Qubes Core Stack has been designed and optimized to bring user one desktop system built on top of multiple isolated domains (currently implemented as Xen Virtual Machines, but in the future maybe on top of something else), rather than for management of service-oriented infrastructure, where the services are largely independent from each other and where the prime consideration is scalability.

The Qubes Core Stack is Xen- and virtualization-independent, and should be easily portable to any compartmentalization technology.

In the upcoming article we will take a look at the updated device and new volume management in Qubes 4.0.

Qubes OS 4.0-rc1 has been released!

Mon, 31 Jul 2017 00:00:00 +0000

Finally, after years of work, we’re releasing the first release candidate for Qubes 4.0!

Next Generation Qubes Core Stack for better integration

No doubt this release marks a major milestone in Qubes OS development. The single most import undertaking which sets this release apart, is the complete rewrite of the Qubes Core Stack. We have a separate set of posts detailing the changes (Why/What/How), and the first post is planned to be released in the coming 2 weeks.

This new Core Stack allows to easily extend the Qubes Architecture in new directions, allowing us to finally build (in a clean way) lots of things we’ve wanted for years, but which would have been too complex to build on the “old” Qubes infrastructure. The new Qubes Admin API, which we introduced in a recent post, is a prime example of one such feature. (Technically speaking, we’ve neatly put the Admin API at the heart of the new Qubes Core Stack so that it really is part of the Core Stack, not merely an “application” built on top of it.)

There are many more benefits that the new Core Stack brings besides the Admin API. Just to name a few that might be most visible to the user or admin:

Simpler to customize and more flexible Disposable VMs,
More flexible and expressive (qrexec) policy definitions,
Flexible VM volume manager (easy to keep VMs on external drives, or in memory-only),

… and many more! The new Core Stack also brings lots of simplifications for developers of Qubes-specific apps and services. Again, we plan to publish posts about all these cool new features in the coming weeks and months.

One last important comment is that all the work we have done in this area has been Xen-agnostic, aligned with our long-stated goal to make Qubes easily portable between different VMMs (hypervisors) and even non-VM-based systems, such as container-based ones.

Fully virtualized VMs for better isolation

Another important change in this release (this time Xen-specific) is that we have ditched para-virtualized mode and embraced fully-virtualized mode for Qubes VMs. The reason for this move has been entirely security-related, as explained here and here.

Originally, we planned to utilize the PVH mode of virtualization, which combines the benefits of processor virtualization technologies (VT-x and EPT), allowing for simpler code in the hypervisor, thus improving security, with paravirtualized drivers for better performance and improved security due to simplified interfaces to virtualized devices. Even though we have long been using isolated stub domains to keep device I/O emulators outside of the TCB, these stub domains themselves run in PV mode, which we are now moving away from.

Sadly, due to the Linux kernel still not fully supporting this PVH mode (specifically problems with booting the kernels in this mode), we decided to go with the HVM-based VMs for this rc1 release. We plan to switch to full PVH either in the later rc-releases, or in 4.1, depending on the progress of PVH support in the Linux kernel.

Also, as an additional last-minute issue, we discovered that PCI pass-through does not work that well on some systems when using HVM virtualization. Typically this affects USB VMs and only on some systems. Nevertheless, as a precaution, in the default installation we decided to switch the mode of virtualization for these VMs back to PV mode. (The new Core Stack allows one to do this with the flip of a ~~switch~~property :). Here our rationale is that it’s still much better to have PV-based isolation for USB VMs rather than not having USB controllers isolated at all! Again, we anticipate this will be resolved in the upcoming rc-releases.

New approach to UX/UI for better integration

In Qubes 4.0 we also decided to redesign the User Experience (UX) a little bit. Aligned with our long-term vision to hide as much of the Qubes internals from the casual user as practically viable, we made a bold move and… removed the Qubes Manager altogether!

Instead, we believe it makes more sense to utilize as much of the infrastructure already built by professional UX designers as possible. Consequently, most of the Qubes persistent configuration (creation of new VMs, changing their settings as well as the global ones) is accessible through the standard application menu aka “Start Menu”. In addition, we wrote two tiny widgets, which should work with most desktop environments compatible with Qubes (currently this list includes the default Xfce4, the once-default KDE, the community-maintained i3, and awesome). These widgets are used to show live info about the running system state, such as which VMs are currently running, their memory usage, as well as which devices are available to connect to different VMs (and yes, now it is possible to connect USB devices using the GUI, a long requested feature by many of our users).

Advanced Qubes users will surely appreciate, on the other hand, the much more flexible and powerful qvm-* tools, such as the completely rewritten qvm-ls and qvm-prefs, to name just two (again, more on them in the upcoming posts).

Better compatibility and all the rest

Besides the above, there have been lots of other improvements and bug fixes compared to the 3.2 release. We list most of them in the release notes.

Perhaps one worth singling out here, in the context of hardware compatibility, is the upgrade of the default dom0 distribution to Fedora 25. (Before we decompose dom0 into separate GUI and Admin VMs, which we plan to do in 4.1, the dom0 distribution determines how well the GPU is supported.)

Summary

Qubes 4.0 is a significant milestone on our roadmap to implement a reasonably secure desktop/client OS based on the “Security by Compartmentalization” principle (using “Explicit Partitioning Model”, in contrast to the recently popular “Sandboxing Model”).

This is the first release candidate of a largely rewritten complex system, and no doubt early adopters will discover some rough edges here and there. Despite our increasingly sophisticated automatic testing infrastructure, this is simply unavoidable. Consequently, if you want to use Qubes for production, stick to Qubes 3.2 until we release the stable version of Qubes 4.0.

But if you would like to start learning and experimenting with the advanced new features that 4.0 brings, such as the Admin API, or would like to help us reach a stable 4.0 more quickly, or you’re just curious, or want to show off to your friends what a bleeding edge system you have, then please do so and go straight to the [download page]!

On behalf of the whole Qubes OS Core Team,

joanna.

Introducing the Qubes Admin API

Tue, 27 Jun 2017 00:00:00 +0000

This post starts the “cool things coming in Qubes 4.0” series and focuses on what we call the “Qubes Admin API.” This should not be confused with Qubes Salt Stack integration, which we have already introduced in Qubes 3.2.

High-level overview

Let’s start with a high-level architecture picture of how the Admin API fits into the Qubes OS architecture:

As we can see, the main concept behind the Admin API is to let select VMs preform various select administrative functions over the Qubes OS system.

If this idea scares the hell out of you, then, my dear reader, we’re on the same side. Indeed, if we’re not careful, we can use the Admin API to shoot ourselves in the foot. Moreover, it might look like we’re actually adding complexity and enlarging the amount of trusted code (TCB) in Qubes OS. All good intuitions. But below I argue that the opposite actually holds, i.e. that the Admin API allows us to actually shrink the amount of trusted code, simplify trust relationships in the system, and ultimately to improve the overall security at the end of the day. It’s a bit like comparing SSH to Telnet. Admittedly, at first sight, the SSH protocol has much more complexity than Telnet, yet no one questions today that SSH is actually significantly more secure than the much simpler Telnet.

So, let’s first quickly look at examples of why we have introduced the Admin API, i.e. what problems it helps to solve and how.

Management VMs

For Qubes OS to become suitable for use in large organizations and/or corporate environments, it inevitably must become remotely manageable by entities such as corporate IT departments. There are, of course, many ways to implement this, but most would punch too many holes in the Qubes security model. For example, if we wanted to run some management agent in dom0, this would not only open up possible ways of attacking the whole system by exploiting potential bugs in the agent itself, but it would also require us to allow networking in dom0, exposing it to a number of additional attacks.

The Admin API solves this problem elegantly without requiring network access to dom0, and we show exactly how below.

Additionally, the Admin API nicely complements our existing Salt Stack integration. While the latter is perfect for pre-configuration at install time, the former is ideal for ongoing system maintenance, monitoring, and on-demand provisioning of VMs.

Last but not least, we’ve designed the Admin API with the goal of allowing very strict “need to know” (and “need to do” for that matter) rules. This means that it should be possible to have admin roles (implemented as specific VMs) that would be able to e.g. provision and manage a user’s AppVMs, but not be able to read the user’s data! Of course, this is more tricky than it might seem when we look at the diagram above, and I discuss some of the catches and solutions below. We hope this will pave the way for organizations to embrace the idea of non-privileged admins.

The GUI domain

In the current Qubes architecture we’ve combined two different subsystems: 1) the (trusted) GUI subsystem and 2) the Admin stack. They’re both in the same VM: the almighty dom0.

This has somewhat made sense, since the GUI subsystem, which comprises the Window Manager, X server and graphics drivers, must be trusted anyway. After all, the GUI subsystem “sees” all your sensitive data, and it can also mimic any user actions by injecting keystrokes and/or mouse movements.

Also, in contrast to traditional desktop systems, the Qubes GUI subsystem is very well isolated from the external world (user apps, USB devices, networking, etc.), which means there is very little chance of compromising it from the outside.

Yet, there are several good reasons to move the GUI code away from the Admin VM (i.e. dom0) to a separate, less trusted and less privileged VM:

The possibility of efficiently separating the user and admin roles. This is somewhat complementary to the admin de-privileging mentioned above and discussed in more detail at the end of this post.
Protection against hypothetical attacks compromising the GPU via DP/HDMI ports.
The option to use a much smaller, less bloated, more difficult to backdoor-by-the-vendor (ideally reproducibly buildable) distro for dom0.
More freedom in choosing the software stack for the GUI: the actual OS (e.g. Windows instead of Linux?); more recent, less trusted GPU drivers (e.g. proprietary instead of from the Linux tree); more choices with regard to desktop environments and window managers; and an easier upgrade cycle for this whole stack.
Reliable logging, which can be separated and protected from both user and admin interference.
The possibility of implementing a bunch of cool features, which otherwise (i.e. if the GUI were in dom0) would be too risky to even consider, such as:
- Exposing the GUI domain through VNC or RDP to other computers, allowing for screen sharing or a Qubes network appliance.
- GPU/OpenGL virtualization for AppVMs.
- Perhaps even Qubes-as-a-Service running in the cloud?

I should note that, as of Qubes 4.0, we still don’t have a separate GUI domain, but we plan to implement this in the upcoming 4.1 release (and some work has already been done here, in fact). The Admin API will be instrumental in this undertaking.

Safe third-party templates

One of the “bottlenecks” we’ve been experiencing recently with Qubes is that there is a growing number of people interested in creating and maintaining various, more-or-less exotic TemplateVMs for Qubes. But there is a problem with allowing (untrusted) third parties to maintain Qubes templates: the templates need to be installed in dom0.

The primary “carriers” for bringing things into dom0 are digitally-signed RPMs. Of course, the digital signatures are always checked by code in dom0 before the RPMs are processed. While this is an important checkpoint, let there be no misunderstanding: this cannot do anything to stop malicious RPMs if the author has signed, either intentionally or accidentally, a malicious RPM. And RPMs can be malicious very easily, as they contain lots of wrapping scripts, such as pre- and post-install scripts. We have thus been always very careful with enabling third-party repository definitions in dom0.

The Admin API comes to the rescue. You can now process the template’s RPM (check signature, unpack, execute scripts) within a normal, possibly untrusted VM (even a Disposable VM), and then use the Admin API to request installation of the template files (not the RPM!) in dom0. The difference is huge, because now we’re just asking the Qubes Core Stack to use the content of an opaque file, i.e. the VM image, to fill a newly-created LVM volume (which is done by executing dd in dom0, so without parsing the untrusted volume content), instead of performing a very complex package installation operation (i.e. rpm -i some-community-template.rpm). (In Qubes 4.0, we switched to using LVM storage by default for all VM images; more details in an upcoming post.)

Needles to say, not every VM would be able to request template installation in dom0. For this to work, the qrexec policy would need to allow the set of Admin API calls needed for this (see below for practical examples).

Also, let’s be clear: if the third-party template maintainer turns out to be malicious (or the computer on which she builds the template turns out to be compromised), then the user will end up with a compromised template, and each and every AppVM based on or created from this template will be compromised also. But the whole point is that other AppVMs, those based on other templates (e.g. the built-in ones) won’t be compromised. Furthermore, the user might, in some cases, prevent the compromised AppVMs from doing any significant damage, e.g. by using the Qubes firewall to limit their ability to leak out data, or even by running the AppVMs completely disconnected from all networking.

Sandboxed Paranoid Backup Restores

Very similar to creating a new template is a backup restore operation. And just as we want to allow less-trusted third-parties to provide templates, in the case of backups we want to be able to restore backups made on a potentially compromised machine (e.g. an older version of Qubes OS that contained a Xen bug vulnerability or perhaps a backdoored BIOS) without compromising the rest of our new system. This concept, known as Paranoid Backup Restore mode has been described in more detail in a separate post.

As described in that post, one of the problems with the present implementation of the Paranoid Backup Restore Mode is the need to parse the complex backup format. And this very action can now be done from within a Disposable VM using the Admin API.

Policing the Admin API (and catches!)

Because the Admin API has been implemented on top of qrexec, it can be policed using the standard qrexec policy.

This might sound easy, but, as usual, the devil’s in the details. Take a look again at the high-level diagram above. We see that, not surprisingly, all the Admin API-related qrexec services have the same destination: dom0. While this allows us to make somewhat meaningful policies (e.g. mgmtvm1 can invoke admin.vm.List and admin.property.List, but not admin.vm.Create), this is far from what one might like to have. Indeed, we would like to say that a given VM can invoke certain Admin API calls only for specific AppVMs and cannot touch other sets of AppVMs.

To make this finer-grained policing possible, we need a simple trick: the destination argument for all Admin API calls should be interpreted on a slightly higher level than the qrexec protocol level. Our qrexec policy allows us to easily achieve this approach, thanks to the target= keyword used in the policy for all of the Admin API calls, as shown below. This also explains why the Admin API table has a column titled “dest”.

We can thus easily express policies for the Admin API with much more granularity and flexibility. For example:

$ cat /etc/qubes-rpc/policy/admin.vm.Create
mgmt-corpo  $adminvm    allow

$ cat /etc/qubes-rpc/policy/admin.vm.property.Set+netvm
mgmt-corpo  $tag:created-by-mgmt-corpo allow,target=$adminvm

The first rule above allows the VM named mgmt-corpo to create new VMs in the system. Each such VM will automatically get tagged with the created-by-mgmt-corpo tag. We can further refer to this tag in the policy, as shown in the second rule above, to allow said management VM to operate only on the VMs that it created. This rule allows the management VM to change the netvm property on any of the VMs that it has created. Note that we have introduced a new keyword: $adminvm, which is the new preferred way to refer to dom0 in both policies and qrexec calls.

In Qubes, VMs are identified by their names, which the Qubes Core Stack ensures are unique within the system. (This is similar to how paths and filenames uniquely identify files on any filesystem). The use of names, instead of GUIDs, makes it easier for humans to write and audit policies. This means that if the user decides to rename the management VM, the above example rules will need to be adjusted. Luckily, there is no Admin API call to request the renaming of the VM. (We specifically block setting of the name property via the Admin API).

Indeed, in Qubes 4.0 we have decided to make the VM’s name an immutable property of the VM object. Once the user creates a VM, the name is set in stone.

About the only way to effectively change the name of a VM, starting from Qubes 4.0, is to perform the clone operation. It’s important to note that the clone operation in Qubes 4.0 differs from Qubes 3.x and earlier. In 4.0, it might be more accurate to say that we’re cloning the VM’s volume, rather than cloning the entire VM. This is because the new clone operation can only be performed between two already existing VMs, and the operation requires two steps realized by two separate calls: admin.vm.volume.CloneFrom and admin.vm.volume.CloneTo. The first call is typically called by the VM that manages the source VM, i.e. the one whose volume we want to clone, and the second call is expected to be called by whatever VM manages the destination VM (of course these might be the same management VM, e.g. the GUI domain in a single-user, unmanaged system).

This ensures that one management VM (perhaps one running some corporate management agent) cannot impersonate one of the VMs managed by another management VM (perhaps the user-operated GUI domain).

Simple monitoring VM demo

Alright, enough theory and hand-waving. Let’s now get our hands dirty with some command-line, shall we?

All the examples below have been run on a pre-rc1 release of Qubes 4.0, but they should work just the same on the “final” 4.0-rc1, which is expected to be released “very soon” (TM) after the publication of this post.

First, let’s start with something very simple:

[user@dom0 ~]$ qvm-create --label yellow test-mon

Edit /etc/qubes-rpc/policy/admin.vm.List and add the following two rules:

test-mon $adminvm allow,target=$adminvm
test-mon $anyvm allow,target=$adminvm

The first rule is required to get a list of all the VMs in the system, while the second is required to query about the state of each of those VMs. Note that the $anyvm keyword does not include dom0 (aka the $adminvm).

If we also want to allow the monitoring VM to query various properties of the VMs in the system, we should also add the second rule to the admin.vm.property.Get file (or else qvm-ls run in our VM won’t display any more info besides the name, type, and power state of the other VMs in the system).

Likewise we should also allow the VM to query the list of labels defined in the system, by inserting the first rule above (i.e. test-mon $adminvm allow,target=$adminvm) in the admin.label.List policy.

Now, let’s start our new monitoring VM, switch to its console, and run qvm-ls:

[user@dom0 ~]$ qvm-run -a test-mon gnome-terminal
[user@test-mon ~]$ qvm-ls

Above we 1) created a new VM named test-mon, 2) added a qrexec policy to allow this VM to issue the admin.vm.List Admin API call, 3) started the VM, and finally 4) ran qvm-ls inside it. It should show the same output as when you run qvm-ls in your dom0 console!

One cool thing you can use this for is to provide IP address resolution for Qubes AppVMs in some ProxyVM, such as sys-firewall. Then, when running a network monitoring tool, such as tcpdump, wireshark or iftop, you would get nicely resolved IP addresses into Qubes VM names. :)

Take a look at the Admin API calls table to see what other calls your monitoring VM might want to use. We might add more in the future, e.g. to allow measuring (i.e. hashing) of various things, such as the policy used, template root images, etc.

Simple management VM demo

Now, let’s move on to something more serious. We shall now create a real management VM:

[user@dom0 ~]$ qvm-create --label green test-mgmt

We will first create a rule to allow our VM to create new VMs in the system by adding the following rule to the admin.vm.Create.AppVM policy:

test-mgmt $adminvm allow,target=$adminvm

We can actually be even more picky and add this rule to the file: admin.vm.Create.AppVM+fedora-25, which would allow the creation of templates based only on the (already installed) template named “fedora-25”. Similarly, we could allow only select VM properties to be read or modified. You can read more about qrexec policy arguments and policing them here.

Additionally, we will add the following rule to /etc/qubes-rpc/policy/include/admin-local-rwx:

test-mgmt $tag:created-by-test-mgmt allow,target=$adminvm

This rule is where the Admin API really shines. As previously discussed, it allows our VM to manage only those VMs that it previously created, and not any other VM in the system!

Notice the file in which this rule was placed: include/admin-local-rwx. This is a special include file. Before we explain what these new files are and how they work, let’s briefly discuss how Admin API calls are classified in Qubes 4.0.

Each Admin API call can be classified as having either global or local scope. Global scope means that it can somehow affect (or query) the whole system state, while local scope means that it can only operate on some subset of the VMs (e.g. those granted via the created-by- tag mentioned above). If a call seems like it fits into both the local and global scope categories, then it is classified as a global call.

Additionally, each call can be assigned one or more of these three attributes:

R(eading)
W(riting)
X(ecuting)

The R-calls do not affect the state of the system. They only return information about the system (or specific VMs). Both the W- and X-calls affect the state of the system, but while the W-calls change the persistent state (e.g. create new VMs, adjust properties of the VMs, or change firewall policies), the X-calls affect only the volatile state (e.g. start/stop VMs).

Now back to our special include files. We provide 4 predefined includes (all in /etc/qubes-rpc/policy/include/):

admin-global-ro – for all R-calls with global scope
admin-global-rwx – for all RWX-calls with global scope
admin-local-ro – for all R-calls with local scope
admin-local-rwx – for all RWX-calls with local scope

Each individual policy file (e.g. /etc/qubes-rpc/admin.vm.List) includes one of the above files at the beginning. Thus, by adding a rule to one of these default include files, we conveniently apply the rules for all the calls mentioned above (unless the user modifies the specific per-call policies not to include these default files for some reason). In addition, both admin-local-ro and admin-global-ro also include admin-local-rwx and admin-global-rwx, respectively.

Before we test things, we need to allow two more R-calls, which are needed due to a temporary limitation of the current implementation of the qvm-* tools, which always attempt to acquire the list of all the VMs in the system. So, we need to either grant access to all the global R-calls (note that we added the rule above to admin-local-rwx, so this time we would also have to add the rule to admin-global-ro), or we need to be more precise by selectively allowing only admin.vm.List and admin.label.List calls to $adminvm:

test-mgmt $adminvm allow,target=$adminvm

In upcoming releases (beyond 4.0-rc1) we plan to remove this limitation, allowing for the possibility of management VMs that cannot get a complete list of all the VMs in the system.

Now, let’s see how our new management VM works:

[user@test-mgmt ~]$ qvm-create --label green managed-work
[user@test-mgmt ~]$ qvm-create --label red managed-research
[user@test-mgmt ~]$ qvm-create --label green managed-vpn
[user@test-mgmt ~]$ qvm-prefs managed-work netvm managed-vpn
[user@test-mgmt ~]$ qvm-prefs managed-research netvm sys-whonix
[user@test-mgmt ~]$ qvm-ls
[...]

With the commands above — all executed inside test-mgmt — we did a few things: 1) we created three AppVMs: managed-work, managed-research, and managed-vpn, 2) we set managed-vpn as the network provider (“NetVM” in Qubes parlance) for managed-work, and we also 3) set the system’s default sys-whonix as a network provider for the other VM.

There are some potential problems with this solution, though. First, sys-whonix is not really managed by us, but rather by the user (or perhaps by some other management VM; we — the test-mgmt — can’t even tell due to the strict policy we used). Second, all the AppVMs we created above are based on the default template, which is also not managed by us. Finally, we would also like to be able to pre-configure some VMs for the user, e.g. configure the VPN client, firewall rules, pre-install some software in the VMs, etc.

One way to solve all these problems in one step is to have our management VM bring its own template(s), which will be used to create the AppVMs that it will subsequently be managing. This is indeed very simple:

[user@test-mgmt ~]$ wget https://repository.my-big-organization.com/qubes-work-template-1.0.0.rpm
# Check the digital signature perhaps? :)
[user@test-mgmt ~]$ sudo dnf install qubes-work-template-1.0.0.rpm
[user@test-mgmt ~]$ qvm-prefs managed-work template qubes-work-template
[user@test-mgmt ~]$ qvm-prefs managed-research template qubes-work-template

The template installation step, which is done inside the management VM (rather than in dom0) works because its post-installation scripts automatically execute commands such as qvm-template-postprocess (which, in Qubes 4.0, replaces the qvm-add-template command along with a bunch of other tasks). These, in turn, are piped over Admin API calls to dom0 (provided that the qrexec policy allows for it, of course). Actually, to support the template installation fully, one would also need to allow the management VM to request the qubes.PostInstall call to the newly-installed TemplateVM. (Note this call is not part of the Admin API, because it is implemented by the template itself.) This call wraps many of the actions that must be performed after the template gets installed, such as determining whether it runs the Qubes GUI agent and allowing it to properly set features for the newly-installed template (more about this feature in an upcoming post).

The same could be done, of course, for the templates used by managed-vpn and sys-whonix. (Presumably, these would be different templates from the “work” template, but they don’t have to be.) These custom templates would likely already bring most of the pre-configuration needed for the user. However, if some additional work was needed, e.g. putting in some user credentials for the VPN, this could be achieved using e.g. the standard qubes.VMShell service. This service would, however, need to be specifically enabled, as by default Admin API calls do not imply that the management VM has the ability to execute arbitrary commands in the VMs it manages.

A better alternative would be to use the upcoming qubes.InstallPackage service, which we’re planning to introduce shortly after releasing Qubes 4.0. Note that this will not be an Admin API call, because it’ll be serviced entirely by the destination VM without any help from the Qubes Core Stack (other than policy enforcement).

The attentive reader will be quick to point out that package installation in a VM is really no different, from a security point of view, from arbitrary code execution, but I argue below that there are important differences that can be used to limit the (ab)use of admin power.

Towards non-privileged admins

As more and more of our activities move to the world of computers, it’s becoming increasingly worrying that there is a privileged group of admins who have nearly unlimited power over the (digital) lives of us mere mortal users. Interestingly, it is not only users who do not feel comfortable with omnipotent admins, but also the organizations who hire the admins, and even some of the admins themselves. (E.g., some admins may not want the legal liability that comes along with having that power.) Hence, the reduction of admin privileges seems like one of the rare cases in which the interests of individual users and large organizations align.

Let’s now think about how can we use the Admin API to make Qubes less prone to admin abuse. First, let’s observe that none of the Admin API calls grant direct code execution inside any of the managed VMs or dom0.

So, how can we configure and provision the managed VMs then, if not with qubes.VMShell? As already mentioned above, there are a few other, better, options:

Bring a pre-configured template.
Provision secrets using a dedicated service. In the most trivial case, just qubes.FileCopy (typically combined with a pre-configured template, which expects, e.g. VPN credentials, to be copied into: ~/QubesIncoming/mgmt/vpn-credentials/.
Request the qubes.InstallPackage service mentioned above (which can only trigger the installation of a package from an already-defined repository of the VM).

At the end of the day, of course, admins choose and install software for the user, so they can always decide to install malicious software. But it should be clear that there is a difference between the unconstrained ability to simply read user data or execute arbitrary commands in the user’s VM vs. the need to deploy specially backdoored software, potentially also bypassing auditing and signature checking policies. The main goal is to prevent the admin from direct attacks that can copy (i.e. steal) user data or inject backdoors in an unnoticed way.

Additionally, we should point out that, while currently admin.vm.volume.Import does not enforce any signature checking on the imported VM volumes (e.g. for newly installed templates), this could be relatively easily implemented via a dedicated Qubes Core Stack extension (more about extensions in the upcoming post).

Tricky backup operations

Speaking of copying VM images (notably, their private volumes), we cannot avoid mentioning backups. After all, the very essence of a backup operation is to copy user data and store it somewhere in case things go wrong. This operation is, of course, in the domain of admins, not users. So we had to figure out how to allow making backups without revealing user data.

Part of the solution to this problem is something we have long had on Qubes OS. Our unique backup system allows us to write backups into untrusted VMs (and, e.g., copy them to an untrusted NAS or USB device). This is possible because we perform the backup crypto operations in dom0, not in the destination VM.

However, when separating the roles of admins and users, we need a way to provide the backup passphrase in such a way that the management VM (which ordered the backup creation) will not be able to sniff it.

The way we decided to tackle this problem in the Admin API is through the introduction of backup profiles: simple files that must be created in dom0 and that contain the list of VMs to backup, as well as the command to obtain the passphrase to encrypt the backup. The command might be, e.g., a trivial qrexec-call to some VM (over which the admin doesn’t have control!), perhaps extracting the passphrase from some Hardware Security Module.

Towards sealed-off dom0

Ideally, we would like to de-privilege both the admin and user roles enough that neither can interfere with the other. This means, e.g., that the admin will not have access to the user’s data, while the user cannot interfere with the admin’s policies. Of course, at bottom, there will still be dom0 with its ultimate control over the system, but perhaps it could be sealed off in such a way that neither admins nor users can modify system-wide policies, VM images, or dom0 software.

One approach to this would be to use a digitally-signed filesystem in dom0 (in additional to an encrypted filesystem, which we’ve been using since the beginning). When (or if) combined with a hardware-enforced root of trust, such as Intel Boot Guard, Intel TXT, or perhaps Intel SGX, this would make it difficult both for the user and for any attacker who might gain physical access to the laptop (e.g. an Evil Maid) to get access to VM images (where user data are kept, or where backdoors might be injected), system configuration (which includes various policies), and other critical files.

The keys to dom0 could be generated during installation and kept by some special admins (superadmins?), split between several of an organization’s admins (and perhaps also the user), or even discarded after initial provisioning. Different deployment scenarios would have different requirements here.

Skeptics might argue that if one has physical access to the computer, then it’s always possible to bypass any sort of secure/trusted boot, or remote-attestation-based scheme. Probably true, but please keep in mind that the primary goal is to de-privilege admins so that they cannot easily steal users’ data, not to de-privilege users. If the user is willing to spend tens or hundreds of thousands of dollars on mounting an attack, then this same user will likely also find other ways to leak the crucial data, perhaps using methods not involving computers at all. So this is mostly about keeping users away from admin tasks in order to allow the smooth operation of both user and admin activities.

Similarly, admins (and developers!) can always insert backdoors, but we want to make it as expensive and as auditable as possible.

All this is planned for beyond even Qubes 4.1, most likely for Qubes 5.0. But we’ve just built the architectural foundation required to implement this new kind of security model for endpoint devices (and perhaps not just for endpoints?).

Power to the (power) users

But what about individual power users? After all, there is a group of people who are not only not interested in having their (endpoint) system managed by someone else, but who actually have an allergic reaction to even the remote possibility of this being an option in their systems. Likewise, any kind of user “lock-down” idea is perceived as taking away their basic freedoms.

The Qubes Project wishes to avoid taking sides in this battle between locked-down computing vs. user-tinkerable computing. Instead, we would like to provide tools that allow both scenarios to be realized, recognizing that there are legitimate scenarios for both approaches.

First of all, it should be clear that Qubes OS, i.e. the software, cannot lock anybody down. This is because the only way to lock down a computer is to have its hardware implement the locking.

Admittedly, the system might cooperate with the hardware to sustain the locking, e.g. by not allowing arbitrary programs to be executed. But with Qubes OS being open source this is never going to be a problem, because those who don’t agree with our decisions regarding role compartmentalization will always be able to build the whole OS from scratch with whatever modifications they desire.

Second, the separation of admin and user duties should be considered on a logical level. In scenarios where the same person plays all three roles – user, administrator, and super-administrator – it is still preferable for Qubes OS to implement this compartmentalized model of roles. In such scenarios, it just so happens that a single person possesses the keys/passphrases to all of the roles.

But then, one might wonder, what prevents employees from using Qubes systems configured this way (i.e. all power to the user) within organizations, and thus bypassing any organization-imposed policies? That’s simple: remote attestation of some sort (Secure/Trusted Boot + TPM, Intel TXT, Intel SGX, maybe even something else…).

So it all comes down to this: either the user “roots” her own Qubes system and has unlimited power (and yet, she might still want to logically separate these powers within the system), or she plays by the rules of some organization (and, at minimum, doesn’t have the keys to dom0, while perhaps ensuring that no one else has them either) and is able to pass the organization’s remote attestation checks and get access to some (organization-provided) data and services.

Summary

The new Qubes Admin API represents a new approach to endpoint management, which has been optimized for symmetric user/admin role separation. Not only is the user prohibited from interfering with admin-enforced policies; the admin is equally prohibited from stealing or otherwise interfering with the user’s data. Furthermore, it is possible to implement multiple mutually distrusting or semi-distrusting management VMs.

The Qubes Admin API has been enabled by the new generation of the Qubes Core Stack, which we’re introducing in Qubes 4.0, and which I will describe in further detail in an upcoming post. Meanwhile, we hope to release Qubes 4.0-rc1 in the coming weeks. :)

Finally, I’d like to point out that nothing discussed in this post has been Xen-specific. This is aligned with our long-term vision of making Qubes work seamlessly on many different platforms: different hypervisors, container technologies, and possibly even more!

Compromise recovery on Qubes OS: individual VMs & full system cases

Wed, 26 Apr 2017 00:00:00 +0000

Occasionally fuckups happen, even with Qubes (although not as often as some think).

What should we – users or admins – do in such a situation? Patch, obviously. But is that really enough? What good is patching your system if it might have already been compromised a week earlier, before the patch was released, when an adversary may have learned of the bug and exploited it?

That’s an inconvenient question for many of us – computer security professionals – to answer. Usually we would mutter something about Raising the Bar(TM), the high costs of targeted attacks, attackers not wanting to burn 0-days, or only nation state actors being able to afford such attacks, and that in case one is on their list of targets, the game is over anyway and no point in fighting. Plus some classic cartoon.

While the above line of defense might work (temporarily), it really doesn’t provide for much comfort, long term, I think. We need better answers and better solutions. This post, together with a recently introduced feature in Qubes OS 3.2 and (upcoming) 4.0, is an attempt to offer such a solution.

We start first with a relatively easy problem, namely recovery of a (potentially) compromised AppVM(s). Then we tackle the significantly more serious problem which is handling the situation of a (potentially) compromised system, including subverted dom0, hypervisor, BIOS and all the other software. We discuss how Qubes OS can help handle all these cases, below.

Digression about detecting compromises

But before we move on, I’d like to say a few words about detecting system compromises, be that compromises of VMs (which actually are also systems in themselves) or the whole physical system.

The inconvenient and somehow embarrassing truth for us – the malware experts – is that there does not exist any reliable method to determine if a given system is not compromised. True, there is a number of conditions that can warn us that the system is compromised, but there is no limit on the number of checks that a system must pass in order to be deemed “clean”.

This means that in many situations it might be reasonable to perform “compromise recovery” for a system or system(s), even though we might not have any explicit indication of a compromise. Instead, the only justification might be some gut feeling, some unusual “coincidence” just observed (e.g. sudden spectacular series of successes of our competitor), or knowledge about a fatal vulnerability that we just learnt could have been reliably used to attack our systems. Or maybe you left your laptop in a hotel room and various entrance indicators suggest somebody paid you a visit?

I’m writing this all to actually make two important points:

As we don’t have reliable indicators when to initiate the recovery procedure, it’s desirable for the procedure to be as simple and cheap to perform as possible. Because we might want to do it often, and just in case.
Instead of expecting the recovery procedure to reveal compromises details (and let us do the attribution, and catch the bad guys), instead we might want to shift our goals and expect something else: namely that once we performed it, we will start over from a clean system.

Handling AppVM compromises

Let’s discuss first how can we deal with a (potentially) compromised AppVM (or some system VM).

How come a VM might become compromised? Perhaps because of some bug in a Web browser or email client, opening a maliciously prepared PDF file, installation of some backdoored software, or gazillion of other potential issues that might lead to a compromise of an AppVM.

It’s worth reiterating here that the Qubes OS security model assumes AppVM compromises as something that will happen on a regular basis.

I believe that, even if all the sharpest vulnerability researchers could spend all their lives auditing all the application software we use, and finally identify all the relevant bugs, this process would never catch up with the pace at which new bugs are being added by application developers into new apps, or even into newer versions of the apps previously audited.

Additionally, I don’t believe that advances in so called “safe languages” or anti-exploitation technology could significantly change this landscape. These approaches, while admittedly effective in many situations, especially against memory-corruption-based vulnerabilities, cannot address other broad categories of software vulnerabilities, such as security bugs in application logic, nor stop malicious (or compromised) vendors from building backdoors intentionally into their software.

Reverting AppVM’s root image back to a clean known state

Qubes OS offers a very convenient mechanism to revert root filesystems of any AppVM back to a known good state. All the user needs to do is to restart the specific AppVM. This trick works thanks to the Qubes OS template mechanism.

It’s difficult to overestimate convenience and simplicity of this mechanism. Something strange just happened while browsing the Web? PDF viewer crashed while opening an attachment? File manager (or the whole VM) crashed while navigating to some just-downloaded ZIP-unpacked directory? Just reboot the AppVM! If you’re slow at clicking, need to shutdown and start a heavy application like a Web browser, and your CPU is of moderate speed, the whole operation will take about 30 seconds.

The pesky home directory

Reverting the root filesystem of the AppVM to a good known (trusted) state might be a neat trick, but for more sophisticated attacks, especially attacks targeting Qubes OS, it might not be enough.

This is because, besides the root filesystem, each AppVM has also what we call a “private” image (volume). This is where e.g. the content of the home directory (i.e. the user data) is stored. And for obvious reasons the home directory is something that needs to persist between AppVM restarts.

Unfortunately it’s not quite correct to say that the AppVM’s private image contains only “user data”. In reality it might contain also scripts and programs, including such that would be auto-executed upon each new AppVM start. Bash’s ~/.bashrc and ~/.bash_profile, or files in .config/autostart/*.desktop are prime examples, but there are many more. E.g. some configuration files in various Web browser profiles (e.g. prefs.js). And there are also Qubes-specific configuration and customization scripts (found in the /rw/config/ directory). An attacker can use any of these, and probably many more, depending on which other applications the user regularly uses, to persist her code within a specific AppVM. Additionally, an attacker can potentially use bugs in any of the software that is always, or often, run in the AppVM, such as a bug in Nautilus (file manager) or some PDF viewer.

Qubes offers three mechanisms how to deal with this problem:

It is possible to mount one VM’s private image as a volume to another AppVM. One can then run whatever malware scanner or investigative software in that other AppVM. This operation is indeed very easy to do:
```
[joanna@dom0 ~]$ qvm-block -A --ro admin-ir dom0:/var/lib/qubes/appvms/personal/private.img
[root@admin-ir user]# mount -o ro /dev/xvdi /mnt/personal/
```

By using Qubes secure file copy operation. Specifically the user might copy the whole home directory to a new AppVM, and then analyze, verify and start using (the verified) data in a new, non-compromised AppVM:

[user@personal ~]$ qvm-copy-to-vm admin-ir /home/user/
[user@admin-ir ~]$ ll ~/QubesIncoming/personal/user/
total 36
drwxr-xr-x 2 user user 4096 Dec  2 21:22 Desktop
drwxr-xr-x 2 user user 4096 Dec  2 21:22 Documents
drwxr-xr-x 2 user user 4096 Dec  2 21:22 Downloads
drwxr-xr-x 2 user user 4096 Dec  2 21:22 Music
drwxr-xr-x 2 user user 4096 Dec  2 21:22 Pictures
drwxr-xr-x 2 user user 4096 Dec  2 21:22 Public
drwx------ 5 user user 4096 Mar  9 13:10 QubesIncoming
drwxr-xr-x 2 user user 4096 Dec  2 21:22 Templates
drwxr-xr-x 2 user user 4096 Dec  2 21:22 Videos

By having “cleaning” scripts in the root filesystem (so they could execute before the private image gets mounted/used) which, when activated, would sanitize, clean and/or remove most (all?) of the scripts that are in the home directory. Assuming such cleaning script is effective (i.e. does not misses any place in the home directory which would be used to persist attacker’s code), the user might just 1) upgrade the vulnerable software in the template, 2) restart the AppVM. It’s worth stressing that, as the upgrading of the vulnerable software takes place in a TemplateVM (instead of the compromised AppVM), the attacker who exploited the vulnerability previously, has no way of blocking the upgrade.

The first method described above, while most familiar to those used to do forensic analysis on traditional systems, is also the least secure, because it exposes the target AppVM (i.e. the one to which we mount the private image of the compromised AppVM) to potential attacks on the volume and filesystem parsing code.

The second method (Qubes inter-VM file copy) avoids this problem. But, of course, some method of sanitization or scanning of the files copied from the compromised machine will be needed (remember that the compromised AppVM might have sent whatever if felt like sending!). Some users might want to run more traditional, AV-like scanners, while others might want to employ approaches similar to e.g. one used by Qubes PDF converters. Perhaps in some scenarios even better approaches could be used, e.g. verifying code repositories by checking digital signatures.

The third method has not been officially merged into Qubes yet, and it is unclear how effective (complete) it could be in practice, but some discussions about it, accompanied by an early implementation, can be found on the mailing list.

Handling other system VMs compromises

Much easier to recover should be various ServiceVMs, such as USB- and net- VMs. In most cases these do not require any significant user customisations or data (maybe except for saved list of WiFi networks and their passphrases). In this case their recovery should be as easy as just removing and recreating the VM in question.

One could go even further and imagine that all ServiceVMs should be Disposable VMs, i.e. without any persistent storage (no private image). In this case full recovery would be achieved by simply restarting the VM. Thanks to the Admin API that is coming in Qubes 4.0 this approach might become easy to implement and might get implemented in the 4.1 or later version.

Recovering from a full Qubes system compromise

So far we’ve been discussing situations when one or more Qubes VMs are compromised, and we have seen how Qubes’ compartmentalized architecture helps us to recover from such unpleasant situations reasonably well.

But occasionally we learn about bugs that allow an attacker to compromise the whole Qubes OS system. In the nearly 8 years of Qubes OS, there have been at least 4 such fatal bugs, and this justifies having a designated procedure for reacting to such cases. We look at this method below.

Introducing the “paranoid” backup restore mode

In order to provide a meaningful option for users to recover from a full Qubes system compromise, we have introduced “Paranoid Mode” for the backup restore procedure. (The mode is also known as “Plan B” mode.)

The idea behind this is very simple: the user makes a backup of a compromised machine (running the backup creation on this very machine), then restores it on a newly installed, clean Qubes system.

The catch is that this special mode of backup restoration must assume that the backup material might have been maliciously prepared in order to exploit the backup restoring code and attempt to take over the new system.

Naturally, backup encryption and HMAC-based integrity protection becomes meaningless in this case, as the attacker who has compromised the original system on which the backup was created might have been able to get her malicious backup content properly signed with the valid user passphrase. We discuss how we made our backup restoration code (reasonably) resistant to such attacks below.

Now, without further ado, a quick sample of how this new mode is used in practice:

[user@dom0 ~]$ qvm-backup-restore --help
Usage: qvm-backup-restore [options] <backup-dir> [vms-to-be-restored ...]

Options:
(...)
  -d APPVM, --dest-vm=APPVM
                        Specify VM containing the backup to be restored
  -e, --encrypted	The backup is encrypted
  -p PASS_FILE, --passphrase-file=PASS_FILE
                        Read passphrase from file, or use '-' to read from
                        stdin
  -z, --compressed	The backup is compressed
  --paranoid-mode, --plan-b
                        Treat the backup as untrusted, don't restore things
                        potentially compromising security, even when properly
                        authenticated. See man page for details.


[user@dom0 ~]$ qvm-backup-restore --paranoid-mode --ignore-missing -d sys-usb /media/disk/backup.bin
Please enter the passphrase to verify and (if encrypted) decrypt the backup:
Checking backup content...
Extracting data: 1.0 MiB to restore
paranoid-mode: not restoring dom0 home

The following VMs are included in the backup:

----------------------------+-------+---------------------+-------+---------------+--------+
                       name |  type |            template | updbl |         netvm |  label |
----------------------------+-------+---------------------+-------+---------------+--------+
                 {test-net} |   Net |           fedora-24 |       |           n/a |    red | <-- Original template was 'fedora-23'
      [test-template-clone] |   Tpl |                 n/a |   Yes | *sys-firewall |  black |
          test-standalonevm |   App |                 n/a |   Yes | *sys-firewall |   blue |
                  test-work |   App |           fedora-24 |       |      test-net |  green | <-- Original template was 'fedora-23'
           {test-testproxy} | Proxy |           fedora-24 |       |             - | yellow | <-- Original template was 'fedora-23'
 test-custom-template-appvm |   App | test-template-clone |       | *sys-firewall |  green |
               test-testhvm |   HVM |                 n/a |   Yes | *sys-firewall | orange |
    test-template-based-hvm |   HVM |    test-hvmtemplate |       | *sys-firewall |    red |
         [test-hvmtemplate] |   Tpl |                 n/a |   Yes | *sys-firewall |  green |

The above VMs will be copied and added to your system.
Exisiting VMs will NOT be removed.
Do you want to proceed? [y/N] y
(...)
-> Done. Please install updates for all the restored templates.
-> Done.
[user@dom0 ~]$

After the backup restoration we end up with a fresh system that consists of:

Clean dom0, hypervisor, and default system/service VM,
some number of potentially compromised VMs (those restored from the untrusted system),
some number of clean, non-compromised AppVMs.

The user can immediately start using either of the AppVMs, even the compromised ones, without endangering any other VMs.

However, a few things will not get restored when running in paranoid mode, and these include:

Any dom0 modifications, such as the wallpaper and other desktop environment customizations (it’s impossible to sanitize these and securely restore, yet these should be relatively easy to recreate with “a few clicks”),
qrexec policies and firewall rules (what good could be these policies if coming from compromised system anyway?)
all non-basic properties for the restored VMs, such as PCI device assignments.

It should be clear that any attempt to restore any of the above might easily jeopardize the whole idea of paranoid restoration procedure.

Qubes OS vs conventional systems?

It’s worth stressing the difference that Qubes architecture makes here. In case of a traditional monolithic OS, it is, of course, also possible to migrate to a newly installed, clean system. But users of these systems face two challenges:

First, in order to copy the user data from an old, compromised system, one needs to somehow mount some kind of mass storage device to a new, clean system. Typically this would be a USB disk, or a directly-connected SATA-like disk. But this action exposes the clean system to a rather significant attack vector coming from 1) the malicious USB device, 2) malformed partitions or other volume meta-data, and 3) malformed filesystem metadata. All these attacks would require a bug in the clean system’s USB, storage, or filesystem parsing stack, of course. But these bugs might be completely different bugs than the bugs we might suspect the attacker used to infect the first system (as a reminder: we perform all these recovery procedures because we learned about some fatal bug in some critical software, e.g. a Web browser for a monolithic system, or Xen for Qubes).
But even if we neglect the potential attacks discussed in the previous point, still we face a very uncomfortable situation: perhaps we just successfully transferred all the data from the old system, but how could we securely use them now? If we “click-open” any of the files, we endanger to compromise our freshly installed system, for the attacker might have modified any or all of our files after the compromise. This would bring us back to the point we started. A traditional solution would be to restore from an earlier, before-the-compromise-happened backup. Only that, as discussed above, we often lack any good tools to determined when exactly did the compromise happen. Not to mention we would need to sacrifice all the recent work when implementing this strategy.

The Qubes mechanism described in this article has been designed to prevent any of these scenarios.

Under-the-hood of the backup restore “paranoid mode”

So, how have we designed and implemented this paranoid backup restore mode on Qubes OS? To understand our design decision, as well as the current limitations, it is helpful to realize that we want to avoid attacks coming on three different levels of abstractions:

System-level attacks (e.g. malicious USB, malformed filesystem metadata)
Backup parsing-level (e.g. XML parser for qubes.xml file)
Backup interpretation-level (“semantic” level)

Qubes architecture has been designed to prevent level 1 attacks, long before we decided to tackle the problem of malicious backups. This is achieved both through careful compartmentalization, as well as through the actual architecture of the backup system, which assumes that whatever domain (VM) that provides the backup should not be trusted.

Level 2 attacks mentioned above is more problematic. The primary attack surface in case of backup restore procedure on Qubes is parsing of the qubes.xml that is part of the backup and which contains crucial information about the VMs being restored (which template they are based on, which network VMs they should be connected to, etc). In case of normal (i.e. non-paranoid mode) backup restoration, even though the backup file is being served from an untrusted entity (e.g. a usbvm), the attacker cannot control the qubes.xml file, because the whole backup is crypto authenticated. Unfortunately, in the scenario we’re considering here, this protection doesn’t work anymore, as explained above.

In order to properly defend against attacks on the XML parser in the backup restoring code, we need to sandbox the code which does the actual parsing. Yet, it’s somehow problematic how such a sandboxed code could still perform its stated goal or creating the actual restored VMs. Luckily, the new Admin (aka Mgmt) API, which we have introduced in Qubes 4.0, is an ideal mechanism for this job.

Unfortunately this API is not available on Qubes 3.2, which means we cannot easily sandbox the backup parsing code there. In this respect we would need to trust the Python’s implementation to be somehow correct and not exploitable.

Finally, there is Level 3 of potential attacks. These would exploit potential semantic vulnerabilities in backup restoration, such as injecting a malformed property name for one of the to-be-restored VMs in such a way that, when actually used later by the Qubes core code, this might result in the attacker’s code execution. For example some properties are used to build paths (e.g. kernel name for PVMs), or perhaps are passed to eval()-like functions.

In order to prevent such logic vulnerabilities, we have decided to write from scratch special code (SafeQubesVmCollection()) which parses the backup and creates VMs from it. The difference which makes this code special is that it only takes into account some minimal set of properties, i.e. those which we consider safe (for example it ignores PCI device assignments, doesn’t restore firewall rules, etc). Additionally it skips dom0’s home directory and implements other limitations, as already mentioned earlier.

One nice thing about the upcoming Qubes 4.0 core stack and the Admin API is that it is trivial to take this safe backup restoration code and run it in other-than-dom0 VM, e.g. a Disposable VM. Then, assuming this VM will be allowed to request specific Admin API qrexec services (e.g. mgmt.vm.Create.*), everything will work as before. But, this time, the XML parser will run sandboxed within a VM, not in Dom0, as in Qubes 3.2. This is illustrated on the diagram below.

Summary

Qubes architecture provides some unique benefits when recovering from compromised (one or more) AppVMs. These include: 1) easy way to revert back to good known root filesystem for all template-based VMs, 2) ability to safely migrate select data from compromised AppVM to new VMs, 3) easy way to recover from compromised system VMs, such as net- and USB- VMs, and 4) ability to reliably upgrade vulnerable software within AppVMs by performing the upgrade in the template, instead of in the (compromised) AppVM, and then restarting the AppVM.

But even more spectacularly, the newly introduced “paranoid” backup restore mode offers a simple and generic way to recover from full system compromises.

FAQ

I don’t get what’s all the fuzz about here? How is this different from any other OS?

Please (re-)read this section and summary section again.

For full system recovery, is it enough to reinstall Qubes OS still on the same hardware?

If our x86-based computers were more trustworthy, or at least more stateless, it would be enough to just reinstall the OS from scratch and stay on the same machine. Unfortunately, as of today, this might not be enough, because once the attack gained access to dom0/hypervisor, she might be able to seriously compromise the underlying hardware and persist malware in such a way that it can survive further system reinstallations.

Wait, I thought Qubes OS offered protection against these pesky SMM/firmware malware?

Correct. Qubes OS, unlike most other systems, has been designed to keep all the malware away from interacting with the real, sensitive hardware, thus preventing various BIOS/firmware attacks (even if the attacker managed to compromise one or more of the VMs). However, once there is a fatal vulnerability in the core software that is used to implement Qubes security model, e.g. in the Xen hypervisor, then this protection no longer works. Sorry.

Alright, but in practice, do I really need to get a new machine?

Unfortunately no one can provide any good answer to that question. Each user must decide by themselves.

Wouldn’t it be easier to just use Disposable VMs for everything?

Unfortunately, the “Just use Disposable VMs” is not a magical solution for all the security problems. In particular, whenever we want to persist user data/customisations/configuration across AppVM restarts (and we want to do that in majority of cases), the use of DispVMs does not provide significant benefits over traditional AppVMs. In some special scenarios this might make sense, like e.g. for the case of service VMs mentioned above. However, it doesn’t seem feasible to have a generic solution that would be able to selectively copy back and sanitize user files before a DispVM shuts down, for later use of these files in another DispVM. And without sanitization, the solution becomes equivalent to… just using a standard AppVM.

The Invisible Things

Debunking the myth of the dark Web

Deepweb

Early meaning of Darknet

Silk Road

Stop using the term “dark web.

Graphene OS -Hardened security Handbook

Hardened // Security Step 1: Use a different profile for the main system

Hardened // Security Step 2: Setup, disable all unneeded features to increase security.

Hardened // Security Step 3 : Disposable Container Protection Measures

Hardened // Security Step 4 : Setting up an environment like Tails / Whonix to connect to the tor network 100% of the time.

Hardened // Security Step 5 : Daily Browsing/Anonymous

Misperception/actualization clarification:

Glossary:

Announcing Wildland Client v0.1

The client functionality

The Wildland demo forests

What’s coming up next?

The Next Chapter: From the Endpoint to the Cloud

Introducing graphene-ng: running arbitrary payloads in SGX enclaves

Enclave-based computing

Intel SGX challenges and problems

Running whole applications in SGX enclaves

Graphene, graphene-ng and Golem

Demo time!

Future plans and wishes

Qubes Air: Generalizing the Qubes Architecture

Why?

Deployment cost (aka “How do I find a Qubes-compatible laptop?”)

The hypervisor as a single point of failure

How?

Example: Qubes in the cloud

Example: Hybrid Mode

Example: Qubes on “air-gapped” devices

Under the hood: Qubes Zones

Under the hood: qubes’ interfaces

GUI virtualization considerations

Digression on the “cloudification” of apps

Summary

Introducing the Next Generation Qubes Core Stack

Qubes Core Stack vs. Qubes OS

Qubes VMs: the building blocks for Explicit Partitioning Model

Properties, Features and Tags

Disposable VMs redesigned

Qubes Remote Execution (qrexec): the underlying integration framework

More expressive qrexec policies

qubes.xml, qubesd, and the Admin API

The new attack surface?

Summary

Qubes OS 4.0-rc1 has been released!

Next Generation Qubes Core Stack for better integration

Fully virtualized VMs for better isolation

New approach to UX/UI for better integration

Better compatibility and all the rest

Summary

Introducing the Qubes Admin API

High-level overview

Management VMs

The GUI domain

Safe third-party templates

Sandboxed Paranoid Backup Restores

Policing the Admin API (and catches!)

Simple monitoring VM demo

Simple management VM demo

Towards non-privileged admins

Tricky backup operations

Towards sealed-off dom0

Power to the (power) users

Summary

Compromise recovery on Qubes OS: individual VMs & full system cases

Digression about detecting compromises

Handling AppVM compromises

Reverting AppVM’s root image back to a clean known state

The pesky home directory

Handling other system VMs compromises

Recovering from a full Qubes system compromise

Introducing the “paranoid” backup restore mode

Qubes OS vs conventional systems?

Under-the-hood of the backup restore “paranoid mode”

Summary

`qubes.xml`, `qubesd`, and the Admin API