theiostream

Viewing the Brazilian Government’s Budget Data Nicely

It’s been almost a year ever since I did any original posts on this blog. Ever since my fiasco attempting to do an app for my school (there’s a blog post about it) I’ve been spending most of my time dedicating myself to social sciences studies as well as a job I recently found. However, between September 2014 and March 2015 I’ve been developing this side project and just now I realized that it’d be nice to write a blog post about it.

In 2014, Brazil had its most disputed election since 1989. President Dilma Rousseff was reelected with 51% of the votes against senator an former governor of Minas Gerais Aécio Neves. In such a disputed election, it’s no wonder that the debates on television featured many corruption accusations, critiques to the administration as well as a candidate saying gay people should “go away” because they can’t make babies with an anus.

However, something that called my attention more than anything else was the fact that it would always be pointed out that Mrs. Rousseff had not spent as much as was planned on a certain thing; that certain funds destined for certain thing wouldn’t be placed there; that there was a reduction of whatever percent on investments in healthcare, and she’d mention the same claims regarding Mr. Neves’ former administration of the Minas Gerais state. I began wondering where all of this data had come from; how could we, common citizens, check those sorts of things for ourselves. Then began my search for tools to do so.

The first thing I came across is the Yearly Budget Law, which comes bundled with five unreadable PDFs which list all government spendings by ministries, topics, projects etc.

This is, of course, almost unbrowseable. It’d take a specialist journalist long enough to find whatever he’s looking for, and even more to do some basic operations with it, such as comparing this data set with the ones from previous years, or making proportional comparisons with other spendings. However, it’s 2014. No one would have typed this by hand. There had to be a database somewhere, and given that since 2011 Brazil has had a Freedom of Information Law, it should as well be open.

The first thing I came across was the Brazilian Senate’s tool Siga Brasil (literally, “Follow Brazil”), which offers some better browsing functionality: it allows you to search spendings by ministries, or by government programs, or by government actions etc.; lists what was actually spent instead of only what’s designed to be spent and puts things in a nicer table. However, it doesn’t look very nice, is hard to find and navigate, and while it may be a data analyst’s paradise, it’s far from the right tool to allow for all citizens to browse the Federal Government’s budget.

The thing that bothered me most about it aside from those things was the lack of a graphical interface (with charts that actually lets someone intuitively understand spendings), historical comparisons (still lacking), and so on. Newspapers had entire teams to do infographics out of this data, however if, once more, a lone citizen was lost attempting to understand certain spendings that weren’t in the news at the moment he’d be utterly lost.

I would later find out that this NGO called Inesc had thought the same thing and attempted to show Siga Brasil data with a nicer interface, in a website called Orçamento ao seu Alcance, which was also opensource. If I hadn’t found out about it only after having almost finished my entire solution, I might have contributed to it instead of building from scratch. It lacks, however, access to some more detailed data about certain projects that Siga Brasil does offer and the developers chose simply not to show.

However, the second thing I found almost seemed to me like a miracle. Among all of those faults, the Ministry of Planning had created a RDF index of all government spendings that would, some months later, win an international prize for being the best open budget database in the world. It had been around for over a year and no one had yet done anything to work with it? Well, that was about to change, for I was ready to build a website that would work with this data and show everything anyone would want to look for as long as we were talking about government expenses.

The only decent library I found for dealing with this sort of data was Apache Jena, which, incredible as it may be doing what it has to do, has no decent bindings to any language aside from Java. Having never programmed a website or Java (which I came to hate), I googled “simple java web framework” and I found this framework, with which I built a website which served requested JSON content with relevant data from the RDF database. It would be up to the JavaScript side to request it, parse it, and show nice graphs with d3.js, which is also incredible.

After some months of work, here are some of the things it offers:

A bubble chart with proportional spendings for all Brazilian ministries:

A detailed, complete list of pension spendings of an specific type in the Budget:

A historical review of government public internal debt:

A comparison between government spendings in paying the internal debt and all educational expenses:

All of this is accessible through a hierarchical, easily browseable interface, which sadly has yet found no webhost.

However, I just felt it’d be nice to share this after, well, almost one year without posting anything. I hope you like it. It’s opensource at GitHub.

PortoApp: a story

This is a repost after 2 months or so that this post had been removed for private reasons.

In Sep 16, 2013, I pushed the first commit to PortoApp’s GitHub repo. Back then, I wanted to make an experiment on managing sessions on iOS. I shamelessly got chpwn’s news:yc’s authentication model, copied his entire login controller by hand and I logged in at my school’s server.

Then, I implemented a news parser for the school’s website. Then, I made a grades viewer/simulator. Then, I displayed the school’s memos decently (and to do so I had to write a whole ASP.NET view state parser in C). The last pre-release commit was in 23 Apr 2014.

Roughly one month later (today), after finishing a four-hour test, my school called me for a meeting. I met with the Educational Councillor (a somewhat psychologist for student issues), the Educational Vice-Director, the Educational Technology Director and the IT Department Director of the school. They told me I was infringing copyright over their website’s content and had the “challenge to think about how what I did was ethically wrong”. It was a big sermon on how I’d be sued if it was the real world, and how they were being so friendly by talking to me, and so on. They never said it, but they naturally meant that there would be legal consequences for me if I did not remove connecting to their website as a feature in my app. In other words, I should turn it into a generic “grade calculator”.

For a notion on what the app’s like, check out the first section. For the school’s attitude, check out the second section. If you check both out, <3.

Development

I’ve been studying in my school for 8 and a half years now. I love it. Students, teachers, security guys, they’re all incredible. As a social sciences lover, I also find the debate groups and all sorts of that stuff awesome. I see my school as a second home. Well, I’d better like it, since my parents pay approximately 2500 Brazilian Reais a month for me to study there. I’ll not go into details, but as for the administrative side, ever since I’ve been elected class president, I can tell you that it can be inflexible, unjust, and simply a pain to deal with.

Ever since I started programming in 2011, I wanted to create something for my school. A Batch script for grade calculation (which I gave up on), a Windows app (which was never more than a white screen), a Mac app (which had the same fate) and finally, an iOS app. Having no longer inspiration to work on what I’d worked for a good while — jailbreak scene tweaks — I decided I’d make an app to prove my friends (and myself) I could do one. Also to learn more about view controllers and so on, at which only making tweaks can make you rather inexperient. In September 2013, I started coding it.

The first thing I wanted done was the backend authentication. What bothered me most on my school website is that it didn’t persist sessions, so you had to login every time you left the website with horrible 3G when you wanted to see grades on-the-fly. I later figured out this was way more complicated than just the login request for other services (a whole documentation on how the school’s website works can be found on this PDF in Portuguese), but still, as mentioned, the first thing I did was pretty much copy news:yc’s authentication model and controllers (with respective changes) and save the login/password on the keychain so it’d login automatically!

Even though I had some initial authentication, I decided to start my GUI with something that didn’t require anything but loading and parsing page. Therefore, I started with school news parsing. Through XML parsing and other HTML scraping techniques, the result became:

Then the time came to implement the most demanded feature a school app has: viewing grades and doing calculations. The first aspect is done by the school’s website, but not the second. The result was an improvement design-wise and utility-wise for every student. Only with HTML scraping, this was the result:

Then, there came the time to implement viewing of school memos on iOS. Accessing this sort of information in the school’s website is slow, and the catalogue page is trouble for mobile browsers. I thought this would be the easiest thing, and it does seem so when you see the end result:

What I did not know is this particular page could not be HTML-scraped due to its dynamic loading capabilities. Therefore, I had to write an ASP.NET view state parser so I could display any information here contained. Doing this data gathering was, I assure you, not very pleasant, given the scarce documentation on the format. I wrote a blog post about it, anyway. Do take a look!

Lastly, it was time for implementing Services. Here there’s room for playing with your entire-year’s grades (not only individual periods), seeing your class and your photo as saved in the school’s website, and also allowing you to access school webpages without the dreadful task of logging in again and again on a browser… you could just add a link to the app and whenever you selected it it’d bring you right to the page, with authentication!

After some last-minute bug fixes, it was accepted and published in the App Store in the start of May.

The School’s Attitude

The local Educational Technology director (not the entire institutional one) was my IT teacher in 2007. Yeah, back in the day where I didn’t know that Shift+4 was the way to type a $. When, in February, all that was missing in the app was everything about Services, I showed her, my only known contact in the IT department in school back then, my app. She liked it. A lot, I think. Few weeks later, I met with the entire institution’s chief Educational Technology Director and showed her my app. She seemed to have liked it as well.

Then, few weeks later again, I met someone who I supposed was the Director of the IT department, then a guy who worked in their website’s development, my dear local Educational Technology director and a guy who hadn’t anything to do with the whole thing: he worked at a company the school had hired for managing academic content on mobile devices for people the Ninth Grade, which get free iPads for schoolwork. I talked about my app, how it worked, what its features were, showed them the opensource code, and they seemed pretty cool with it. I found out they were developing an app for themselves (which had not even half the features or the good-lookiness of mine… I cannot send a screenshot not to get in trouble, but I can assure I say that very honestly), but they said my app looked incredible and that I could eventually release it if I wanted.

When I’d finished everything in the start of April, I sent the supposed Director of IT an email asking her for written authorization for the app’s released. She didn’t answer for 20 days, so I assumed that she was so indifferent to the app that she wouldn’t care if I just released it anyway. And that’s what I did.

I printed out some pamphlets along with a friend who helped make assets for the app (yeah, all assets are mine, there’s nothing that even states the school’s name or any copy of the school’s logo locally). We distributed it to some classes. During a completely unrelated meeting, I showed my Educational Councillor my app completely casually. She seemed to have liked it as well. Later, she’d ask me to remove the pamphlet I’d pasted in my class’ wall (but ironically not the other classes which she has way more contact with), since due to some inexistent rule in the School Regiment I’m not allowed to do that… but who am I to counter orders from the administration?

During May, I needed to publish two updates to fix major issues (one due to my school’s fault changing something about how their website works; one due to a missing call to -setNeedsDisplay on a view). Outside that, what’s interesting was my school’s reaction to the app.

Until today, after a four-hour test, I am called to the Educational Councillor’s office. I meet the guy who’s actually the IT Director, the same global Educational Technology director who’d so much liked my idea, the local Educational Director and the Educational Councillor, naturally. My dear local Educational Technology Director, who might be the only person to defend me, was not summoned.

The conversation starts with them talking about the ethical use of third parties’ data, and eventually comes to the point where they say in real life, I’d get sued for doing the sort of thing I did to an institution (which would be copyright infringement), and that by asking me to remove my app from the App Store, they were teaching me a lesson. In the end, they said I had the “challenge” to think about how unethical I was. What they wanted me to make was to stop data gathering from their website and turn the app into a generic grade calculator with the above-shown display. Wouldn’t it be then easier to put your grades into an equation on paper; wouldn’t you need to access the school’s website to figure out how much a grade weighs, what’s the value of this and that…

What’s interesting is that I access and display their data, and my app does consist of third-party content that I do not own rights to… but isn’t that what a web browser accesses? Will it sue Chrome because it’s not, as Internet Explorer, entitled to display its content? If I made a browser app for iOS that had a fixed address in their website, would they sue me? I access their content exactly like a web browser would, I get the same exact data in the same way, yet only display it differently… how is that possibly classified as copyright infringement?

They never mentioned my app being paid. Yes, it costs $0.99. They never argumented about how I was profiting over their content and so on. They /just/ used that argument about how a client to a service is somehow illegal and what killed me inside, unethical.

If someone can’t read a copyrighted book properly and you give him glasses, are you committing copyright infringement?

What’s even more interesting is that a certain director who I guess would “protect” me from these things in a certain way was in Germany at the time.

This goes to show that as much as schools attempt to mask the image of a great environment for students to thrive, learn enterpreneurism and so on (mine does that a lot), they’re traditionalists to the point that it doesn’t make any sense. What do they have to lose with my app? How is that classification as copyright infringement even pertinent? They apparently just must have control of anything that has anything to do with the school.

I will not confront them, because as I said, I love my school. But I will never get involved in anything they ask me for.

Rejection from an administration — that’s what I get for doing something students and teachers complimented.

Karl Marx, Net Neutrality and the American Dream

I’ve wanted to write this article for about a week now, but a trip stopped me from doing so earlier. So, I’ve been reading a lot about social sciences recently as well as kept up with news in the coding world, so why not try to mix both in a post? That’s what I’ll attempt to do now with net neutrality, tendencies of the tech industry and so on.

The first thing that should be noted is that this post is based on impressions from articles I’ve read as well as data collected from multiple (pointed out) sources, and not from personal experience working in the tech industry. Whether this should prove good or bad is up to the reader.

The second thing that should be noted is that I don’t have a degree or any formal education in social sciences aside from High School Philosophy/Sociology classes and reading books I find interesting. The same thing applies to programming. I haven’t had any jobs in tech either, only coded some personal projects over the last years which placed together have not given me over a minimum wage for a month in the US.

A Letter from Y Combinator: A Problem

A letter from Y Combinator asking the FCC to enforce nondiscrimination from ISPs (in other words, net neutrality) has the following part:

I was lucky enough to be in the very first round of Y Combinator’s investments and created reddit.com with my college roommate Steve Huffman. We were two recent college graduates with no connections and $12,000 in funding, raised from Y Combinator, building something from a pair of computers in a small rented apartment in Medford, MA. Today reddit is a top 50 website with over 110 million monthly unique visitors – more traffic than CNN.com or NYTimes.com. We lived the American Dream thanks to the open internet and today I’m a partner at Y Combinator.

How? The world isn’t flat; but the world wide web is. It must remain that way.

We need the FCC to keep the level playing field that let me–and so many others–succeed as entrepreneurs. The reason so much innovation and wealth creation has happened in tech over the last decade is that any American with her laptop and Internet connection could build a startup and compete with incumbents (and even beat them) without a team of lawyers and without a large budget to pay for priority from ISPs.

Let me be clear: we need a bright-line, per se rule against discrimination, access fees, and paid prioritization on both mobile and fixed.

The letter mentions the American Dream: being able to start a business from the ground up as many times as it takes and be able to be successful if you have enough energy, creativity etc., an ideology that has persisted in the American mindset through the age of the trusts, the growing difficulty of establishing a small business in the post-depression era, and that has been recently reinvigorated with the pit of opportunities that the Internet offers. Is it the first time in History, though, that there has been a call for the preservation of the American Dream in face of “unfair competition”?

Another point is the proposition that the world isn’t and can’t be “flat”, but that the Internet should remain so. Is this possible?

The last question would be: is net neutrality a way that could assure the continuity of the American Dream when it comes to the Internet?

We’ll deal with these questions in this article.

Older Tendencies

In Capital, volume I, Karl Marx, whose work is (after decades-long anti-communist effort), by most, simply linked to socialism as if there was nothing else to it (dialectical materialist method of analysing history, contributions to social criticism, critique of capitalism etc.), capitalist concentration is at first analysed, even though not much detail is put into it, as Marx himself notes. A brief extract from the section that details it describes briefly why concentration is a tendency:

The battle of competition is fought by cheapening of commodities. The cheapness of commodities demands, caeteris paribus, on the productiveness of labour, and this again on the scale of production. Therefore, the larger capitals beat the smaller. It will further be remembered that, with the development of the capitalist mode of production, there is an increase in the minimum amount of individual capital necessary to carry on a business under its normal conditions.¹

When the first volume of Capital was written and published (1867), trusts and cartels, the first examples of massive capital concentration in History to be seen, were a very small force, so how these massive companies worked was something Marx couldn’t elaborate on. In the first two decades of the twentieth century, Hilferding, Lenin and others would write works on monopolistic financial capital, how it got concentrated and operated, and how it was tied to imperialism²; Max Weber would write on the form of social organisation that rose out of the rationalisation of production: bureaucracies, mainly in Economy and Society (1920). A lot of literature on the topic would follow. In White Collar: The American Middle Classes (1951)³, C. Wright Mills documents how between 1870 and 1940 the role of the small enterpreneur in the economy was diminished, how establishing a successful small business out of the ground got harder and harder, and how middle classes converged from small-property-owners to bureaucratic white-collar employees.

There have been many forms of resistance against this sort of concentration that seems to have gone against the liberal principle of equal opportunities for all, the virtues of free competition and so on, the underlying principles of the American Dream.

The most classic one is what Marx called, in the Communist Manifesto (1848), petty-bourgeois socialism, that is, the socialism of the small artisan, that opposed the growing concentration of production of the factory:

This school of Socialism dissected with great acuteness the contradictions in the conditions of modern production. It laid bare the hypocritical apologies of economists. It proved, incontrovertibly, the disastrous effects of machinery and division of labour; the concentration of capital and land in a few hands; overproduction and crises; it pointed out the inevitable ruin of the petty bourgeois and peasant, the misery of the proletariat, the anarchy in production, the crying inequalities in the distribution of wealth, the industrial war of extermination between nations, the dissolution of old moral bonds, of the old family relations, of the old nationalities.

In its positive aims, however, this form of Socialism aspires either to restoring the old means of production and of exchange, and with them the old property relations, and the old society, or to cramping the modern means of production and of exchange within the framework of the old property relations that have been, and were bound to be, exploded by those means. In either case, it is both reactionary and Utopian.⁴

In summary, it attempts to solve the problems of concentration etc. by returning to an era where it was nonexistent (mercantile capitalism), even though the conditions supplied by that era were the ones that enabled capitalism to develop itself into the system that brought these problems! It also ignores the naturally positive aspect of concentration, namely, advantages in production/distribution organization, which are the reasons for which it happens anyway.

In the aforementioned chapter of Capital, Marx differentiated capital concentration from centralization. The first is a tendency of capitalism through which the growing accumulation of capital with time will lead to production in larger scale; the second is exemplified by the trusts: companies might merge, be acquired etc., leading to larger scale production not as the result of the process of accumulation, rather as the result of deals. In around 1920 anti-trust laws in the US (via acts since 1890) had been around long enough to stop centralization from happening in excess, but the process of concentration grew stronger to the point where, as writes Wright Mills, in 1939, 1% of all US companies employed over 50% of the urban active population.

The 1890 Sherman Anti-Trust Act’s intent was more to defend the right of “fair competition” than provide a healthy market. Wright Mills notes many contemporary politicians that defend the virtues of “fair competition” as well. It all leads him to the conclusion that while the small enterpreneur loses economic power, his ideology, the American Dream, remains alive, and efforts to return to the glorious era of the 1850s where an enterprise could easily be made by anyone remained a goal for many.

The Internet

We’re seeing the exact same phenomenon right now. The Internet was (and still is, to some extent) a complete new sector. A bucket of seemingly endless opportunities. But concentration has arisen, as we can see from the YC letter itself. When it calls for the persistency of a “flat” Internet, of an Internet where everyone can easily start an enterprise, he’s ignoring the tendency that has accompanied capitalism ever since its conception of concentration. It’s positive for development (different subdivisions for developing utility libraries, for different platforms, etc.), many times for the user (who doesn’t like, for instance, what Google does – Docs, Drive, Gmail, social networking, all in one place?).

If we look at computer scenario right now – developers used to get lost in architectures and platforms in the 1980s due to hardware coming from the most diverse manufacturers and operating systems varying a lot. Now we have a handful of hardware manufacturers that sell to a large public and three major operating systems.

US Census data (SUSB) shows this clearly: In 2007, 3% of “Internet Publishing and Broadcasting” companies – 75 enterprises with over 500 employees – had 60% of profits in the sector. In 2011, web search providers were included in this category, yet only 2% – now 108 enterprises – had 77% of the sector’s profits. The total number of companies almost doubled, even though concentration only increased. I don’t know if this company category is correct for all (or even most) cases, but I doubt changing the category will matter much to the results.⁵

The YC letter brags about its companies providing around 3,000 jobs… Didn’t Microsoft just fire 18,000 people without major consequences to its workforce?

Apple, Google and friends’ “labs” have been responsible for a lot of new tech recently. Who’s to say these divisions won’t take the place of today’s startups in innovation? No one’s saying that startups will go extinct or lose all possibility of becoming a giant as Reddit has become, but is it getting any easier to compete in the Internet? Is having a computer and an Internet connection, as says the letter, all that it’d take to compete in a market that tends this way?

Conclusions

ISP priority is something that most certainly accelerates concentration and serves as a palpable example of how larger companies can make competition harder, but is it really what’s causing it? Would net neutrality really stop it? For this article not to end with the same question it started, my opinion is a no. But one’s completely able to take a different conclusion from the arguments presented here – or bring up a different set of data.

And as for the second and unanswered question, I don’t think that we can merely think that the Internet is split from capitalism as an economic system to the point that it can remain “flat” while the rest of the system doesn’t. Thinking that might be another symptom of American Dream mentality – realizing things aren’t so ideal in determined sectors but thinking that in other ones the ideal “fair” competition system can be achieved.

Perspectives

For Austrian economists, as soon as concentrated industries start being bad for the consumer, it’ll fall thanks to an underlying free competition system that’s never gone and stays as strong as ever. So from that perspective, we shouldn’t be afraid of concentration since it’ll be good for everyone, and when it stops being so, it’ll vanish as long as this is not stopped by the State. New technologies might grow from the ground as well and if it’s good enough, it should be enough to defeat large empires.

But if we (and I do) think that this does make competition harder and stops new technology from flourishing, we must think of solutions to this problem. We can’t ensure positive free competition with things such as net neutrality. That’s going a path that has been travelled numerous times in the past, as we could see. The problem’s bigger than that. We must look for a change in the system that’s more than reactionary, more that just making concentration a longer process. What is it? Good question.

Relay

Relay, the most awesome iOS IRC client ever, after two years of constant struggle regarding Gyresoft being lazy and Max Shavrick leaving for a while to visit the math-cave, was released on the App Store.

It’s pretty cool. Download it! :)

An Analysis on iMods

In Critique to the Gotha Program, The Poverty of Philosophy, Anti-Dühring etc. our fellow Marx and Engels have this fun style of criticizing one other philosopher’s text: Split it into paragraphs, and write a whole thesis on why it’s incorrect.

I shall apply it to the new Cydia alternative in the ‘hood: iMods’ About Page, with some English corrections:

We are very excited about what developers have done in the Jailbreak scene on the iOS platform. They have made many tweaks and changes that are able to increase everything from productiveness to visual enhancements.

The first sentence is expected. The second sentence, however, already starts a horrible chain of ideas that will proceed through the article.

Talking about tweaks and changes is certainly redundant, once that as far as its definition of change (increase [of] productiveness [and] visual enhancements¹) goes, tweaks and changes can be put down to tweaks.

This being so, by putting what the jailbreak community has done down to 'the creation of tweaks’, it’s obvious that already in this first paragraph they seem to ignore the big and original purpose behind jailbreaking that justifies its etymology: to give the user of the device hacking freedom, which includes the creation and installation of tweaks.

Not only this statement ignores the philosophy of jailbreaking, it also ignores whole reasons other than tweaks to jailbreak; one could write a book about how jailbreaking allows developers to experiment; security researchers to… research; but let’s keep it down to reasons outside tweaks for why a user would want to do so:

• Apps such as iFile, which allow you to browse files on device, as well as the unlocking of afc2 (which allows clients like iFunBox to access your device files and easily transfer media to your computer without relying on Apple software such as iTunes, iPhoto);

• The installation of SSH, which gets SCP clients to browse/insert files on-device, as well as commandline access to the device, which turns it into a pretty way to hack around and use it as a common Unix workstation.

• A whole catalog of free ringtones/wallpapers on Cydia.

• Apps which due to their purpose and/or obligatory usage of private APIs would never reach the App Store: BatteryDetective, AppInfo, iTransmission, console emulators etc.

First and foremost, in order to install such tweaks [one] requires a jailbreak. The means to allow the installation of such tweaks involves the use of a jailbreak. For those who are technical[ly experienced], Jailbreaking is fine and does not involve too much risk. The average iOS user (users who bought their apple devices for the simplicity and ease of use) view Jailbreaking as a risk that can mess up their phone. The fact that apple voids their warranty does not help. Thus the majority of iOS users do not access these amazing tweaks and features already out of the question for being able to get these amazing tweaks and features.

iMods has no way of fixing this. But it might be worth mentioning that as much as there’s a risk stereotype, let’s say, to the process of jailbreaking, with little research about it you can see that there’s nothing /that/ dangerous about it, and if anything goes wrong there’s a huge support platform around (mainly JailbreakQA, which was preceded in importance by iFans, ModMyi etc. Point is, there was always one.).

And besides, not all jailbreaks evoked this feeling. JailbreakMe, for instance, was always as user-friendly as you can get.

The second problem that has arisen is that within the Jailbreak scene, the majority of users are pirating the software.

By the following quotes, it’s rather obvious that they address piracy of Cydia apps, but it should be interesting to discuss the using jailbreaking as an App Store piracy tool question before we get to that.

Speaking from experience, it is true that most of the ordinary people who I’ve met who are jailbroken did it to install Installous/AppCake/whatever as soon as they got Cydia in hand, and by its 'letting users be free’ philosophy, there was no way this could be stopped.

Yet, due to the philosophy of jailbreaking meaning users being free, there’s nothing much that can be done about that. I’ll expand this argument as this critique evolves.

Users can easily add sources that provide the pirated version of paid tweaks and features (items that developers have spent countless hours for and are asking for a minimal amount of money in return) and receive the exact same app as if they had paid.

Let it be noted that some tweaks implement DRM. Most, though, just don’t care or think Well, pirates will manage to pirate it anyway so there’s no reason I should spoil the user who paid’s experience and there’s no reason they should think otherwise.

The ease of which the Cydia platform allows this to happen does not help the rate of pirating. This is not only a problem with Cydia, even when Rock Your Phone was present, the same problem persisted. During the period of 18 months when Rock was available, only about 500,000 app licenses were sold while 38 billion pirated apps were installed through their software. This means that only 0.0013% of app downloads on Rock were legal. This clearly shows that a lot of potential money for developers was lost, not because it was impossible to stop pirating, but because the ecosystem of the current platforms makes it so readily available.

Let’s ignore this statistical fact, since it’s too vague.

And regarding the “Cydia and Rock’s ecosystem enables easy piracy” argument, let’s get back again to the freeing-your-device argument. As far as jailbreaking is about 'letting users be free’, there’s no reason Cydia/Rock should force you to do otherwise. Cydia does give you a “This repo has been known for piracy” warning so you know what you’re getting into.

Let’s say, though, Cydia (Rock is dead.) said “No, you can’t add this repo, sorry”, what could stop pirates from:

• Creating endless repos with different addresses;
• Using iFunBox for installing cracked IPAs instead of Installous-like software (for App Store piracy);
• Using SSH to install cracked software;
• As a last resort, create a jailbreaking tool which installs a modified build of Cydia (which is opensource!) which doesn’t care for your repos.

That being so, it’s not a matter of Cydia/Rock environment allowing piracy, rather the concept of jailbreaking that makes it so.

The third and most important point, which in some extent goes back to the first problem, is that [packager] softwares such as Cydia, Rock, and Icy are not easy for a person who does not know anything about Jailbreaking to use. They have not been exposed to these softwares and their benefits.

This argument is flawed. There are thousands of resources where one can learn how to mess with Cydia, notably iDownloadBlog published tips for using it once in a while, and did you ever try looking at Cydia’s home tab?!. Let’s see some cool buttons in there:

• Featured [Apps/Tweaks]
• Themes
• [Some featured tweaks, as of now Zephyr, IntelliScreenX, etc.]
• Upgrading & Jailbreaking Help
• More Package Sources
• FAQ
• Different How-Tos
• JailbreakQA / /r/jailbreak links

Seems pretty nice for a starter, huh?

Even if they were, the risks associated with trying to find out which tweaks are effective and safe on their phone (such as the risk of their phone getting stuck in a boot loop) would automatically scare them off.

Boot loops? Are we in an era before MobileSubstrate existed? That’s pretty rare nowadays.

One could argue there are horribly broken extensions which could get you in a boot loop, and that’s why MobileSubstrate 0.9.3997 comes with the awesome feature of Disabling Substrate entirely on boot if the Volume Up button is held.

(And please don’t get me into 'tweaks’ that don’t use MobileSubstrate, if there’s even such thing today.)

And as for finding out which tweaks are effective, the starter can always Google for a “nice tweak list”, read blogs that post about tweaks, or go to Cydia’s Featured/Themes tab which was mentioned above.

If a more user-friendly installation platform existed that made the installation of these features easy for your average iOS user possible, the number of potential customers for these features would be pushed significantly higher.

With the arguments above, there’s nothing much to add.

These potential customers also do not possess the knowledge nor have much desire to obtain tweaks and features illegally. Peace of mind and easy accessibility are of paramount importance to them.

Let’s split this into two pieces:

• [they] do not possess the knowledge […] to obtain tweaks and features illegally. Peace of mind and easy accessibility are of paramount importance to them: So, uh, the reason you want to make the process for installing tweaks more straight-forward is so that users are kept in ignorance about the existence of means of piracy?!

• [they don’t] have much desire to obtain tweaks and features illegally. Peace of mind and easy accessibility are of paramount importance to them: So, people go through finding+adding a new Cydia repo to pirate a tweak because they lack “peace of mind and easy accessibility” to the paid ones? So they get those by downloading pirate software from unreliable sources, which are obviously tougher to add for the so-exemplified ignorant user to add? (well, the second paragraph would most likely argue that it’s easier to pirate in Cydia than it is to legitimately buy stuff… enough said)

We at iMods are excited to say that our closed market platform, aggressive advertising infrastructure, and guaranteed warranty will enable us to target a previously unaccessible demographic that will result in increased revenue and exposure for Jailbreak developers.

• closed market platform: I won’t even comment.
• aggressive advertising infrastructure: Wasn’t ads what everyone who was making other Cydia alternatives (iTweakStore, iDebStore, Icy, Installer) complaining about?
• guaranteed warranty: One should ask repo managers what they think of that. I can’t speak for them on this regard.
• previously unaccessible demographic: I don’t think a pretty store that denies the whole concept of jailbreaking=freedom will attract this so-called previously unaccessible demographic. I wonder how unaccessible this demographic will be once there’s no untethered jailbreak available.
• increased revenue and exposure for Jailbreak developers: Maybe it could increase revenue. Maybe people would be just mad for you using this store to distribute your software and wouldn’t buy your software. Maybe they’d just rather keep using Cydia (see conclusion), which they know by now. Then we wouldn’t be so exposed, would we?

End.

Long story short, this thing would only function properly as an AppStore-like tweak store (which is the only product they view as sellable), which would only achieve its anti-piracy goals by not letting people install Cydia, which would contain pirate copies of the software there distributed (or they could encrypt whatever they distribute and not let it get redist’d by Cydia pirate repos; but it’d eventually get cracked).

I don’t doubt they could convince some jailbreak dev teams cough to drop Cydia cough and install iMods instead cough, but then someone else would just create another jailbreak tool which used the same exploit and installed Cydia. Freedom wins again (with the small cost of piracy.)

If the jailbreak ecosystem were abolished and iMods were to prosper, we’d just enter another jail. A broader one, but still a jail.

Parsing ASP.Net New View State

Introduction

If you’ve been reading this blog for a while, you’ll know two things:

• It’s 2 years old now! (or close)
• Some time ago, I made a small blog post on how to login to my school’s website using HTTP requests.

That post evolved into an app for checking my school’s grades, news, etc, which I’ve been working on for some time and is opensource here.

The time came when I finished my hard work on the grades view controller, and went to getting my school’s Circulares (I can’t think of an English word for it – the papers they give to students to warn them about something), which are available on the website in PDF format, which means I can just crawl them and show them with UIWebView, right?

The fact is, while the rest of my app got data from HTML scraping, this would be particularly hard in this case. You can see why by yourself thanks to this security flaw which allows anyone to check the circulares out here. In fact, you should look here anyway if you want to understand stuff I’ll further talk about this page.

So, this makes HTML scraping hard due to obvious reasons. But I later found out all of the data I needed was in a 450kB-long base64 string inside a hidden input tag in the page called ASP.Net view state.

A long description about view states can be found at this article from Microsoft. Long story short, it’s a way ASP.Net developers find to store data across page postbacks without using databases etc. by sticking an encoded base64 string in a hidden input tag.

So, great! Let’s parse it! Where do I find a parser?

Oh.

As it turns out, I only found two view state decoders around:

• View State Decoder 2 by Fritz Onion, which I had to get from web.archive.org (just throw in this link) since it’s no longer downloadable through standard means (Windows-only);
• An online view state decoder here.

Wait… Windows-only and an .aspx extension? Oh great, these decoders show the decoded output in a nice fashion (which later would be very useful), but they don’t do the decoding themselves… They use a .Net class called LosFormatter (from Limited Object Serialization) to do it.

Wait… What about Mono? It’s cross-platform .Net essentially, right? And they do have a LosFormatter class! Except it’s a whole other format which isn’t compatible with actual .Net.

After a more thorough search, I found this obscure project: ASP .Net viewstate decoder / encoder +. I immediately put my view state string on top of it but as it turns out, it decoded the old view state format (yes, in around 2008 Microsoft changed how they serialized it, hence the blog post title having a new in it). The changelog said it supported the new format, but:

• When inserted, it mis-read the file (either stopped at a newline or at a NUL char);
• All it did to the piece it read was show a hex dump of the base64-decoded output.

So, this left me with three options:

• Create an ASP.Net web application which used LosFormatter to parse my school’s view state and which I could access from my app;
• Fix that view state encoder/decoder;
• Start a decoder from scratch.

The first option was out of question. First, I didn’t want to code up something I had to really rely on for a first C# project; secondly, I don’t have a decent Windows development environment; thirdly, I didn’t want to have a nice enough server to carry a request load. My website is hosted for free by a friend and I don’t know how much it can take.

If I was going to write a parser, I didn’t want to rely on other guys’ code load, mainly if it was going to be mostly unuseful. So I wrote my own view state parser.

How to start, though? I decoded it from base64 using libb64, threw the result in hexdump and stared at it for a while, trying to find patterns. I deduced the 05 byte had to do with strings, but the whole 1f cached indexed string thing confused me, so in the end all I could be sure of is that 0f indicated the start of a pair object.

I then found this pdf. This presentation was just awesome: it showed my thoughts exactly when talking about reasons to understand the format, documented the new view state, but in the end there was no related opensource code (which meant I still had to write my parser), and the only concrete information it gave me was that it had a weird base-128 int32-storage system and in a screenshot showed some of the bytes that introduced information.

But that was good enough. I now had a good idea of how the format worked, how I’d architect a parser and all I had to do now was reverse-engineer the format for real and find the (most) storage types that the pdf author hadn’t found.

Reverse-engineering was done mostly by:

• Installing Visual Studio on my sister’s PC and generating view state data from C# data structures, which allowed me to analyze what was going on in some cases which weren’t there in my school website’s view state.

• Putting selected pieces of my school’s view state inside .Net-based decoders and seeing what it printed.

The result is here.

The format

View state is base64-encoded. Naturally, the first step to decode it is to use a base64 library to decode it. I used libb64, which is simple, small and good enough for my needs.

The output starts with ff 01. Always. The PDF assumes ff is related to UTF-16 encoding, but I could find no evidence confirming or denying that.

The rest of the file works as follows:

type_byte [data bytes which fill it]

That being so, the type bytes are:

• 02: Integer. Its data bytes are a weird base-128 numeric system. The code has documentation enough on it. Note that all lengths of further types (e.g. string length) use this same format.

• 03: Byte. Only one data byte follows.

• 04: Character. Haven’t studied it in detail, will update.

• 05: String. Byte(s) indicating the string length follow and subsequently Unicode bytes which compose the string.

• 14: Array. A type format follows, representing the array members’ type, which is followed by the array length and length times other objects.

• 15: String array. The array length follows, followed by length times a string length and the Unicode bytes composing each string.

• 16: .Net Array List data structure. Followed by a length, and then by length times other objects.

• 18: .Net Hybrid Dictionary data structure. Followed by a Key-Value pair count, which is followed by length times other objects with the <key object> <value object> format.

• 0b: Enum. A type format follows representing the enum type, and then an integer value.

• 0f: Pair. Two objects follow.

• 10: Triplet. Three objects follow.

• 1e: Indexed String. Has the same format as a string. Adds an indexed string to the indexed string cache with index starting in 0 and progressing each created one.

• 1f: Retrieves an indexed string. Followed by a number representing the index of the indexed string to retrieve.

• 3c: Weird Array: A variation of the array which happens in unknown cases. Its format is a type format representing its members’ type, an unknown byte, the array length, another unknown byte, length times other objects which compose the array, separated by the 08 byte.

• 64: Null.

• 65: Empty string.

• 66: The number zero.

• 67: Boolean true.

• 68: Boolean false.

The type format defines a .Net type related to the object which demands it. It has three possible variations:

• 29 <string length> <long description of the type>. This long description includes type version, library of inclusion etc. It adds this type to the type cache, starting on index 4.

• 2a <string length> <type name>. This only includes the type name, usually of types built in System.Web.UI. It adds this type to the type cache, starting on index 4.

• 2b <type number>: Indicates the type through a number. Built-in types are 00(objects), 01(integers), 02(strings) and 03(booleans). Starting on 04, these are types defined with the two previously documented type formats.

That’s it. There are some undocumented aspects here and some unknown stuff, but that’s as far as my parser’s gotten. I’ll update this as soon as I find more stuff or people reach me with more info. But meanwhile I’ll focus on my school app’s development being that now it can parse without any issues the view state there put.

I shall soon release some documentation elsewhere regarding my view state parser. If you read everything up to here, <3.

On jailbreaking

I’d like to announce I am not using the new versions of evasi0n, be they patched or not, unless it decides to go opensource and I can personally check what it is doing to my device and data.

Meanwhile, since I believe it’s not fair for the (few) people who paid for my software to suffer because for this decision of mine, I’ll attempt to get FoldMusic, SearchStuff, etc. working anyway with other means.

I’d welcome any other jailbreak tool to use (probably an addition to rms’ awesome opensn0w?).

But even if either a new tool or opensource (hahaha) evasi0n happens, I won’t be making any new software for jailbroken iPhones (I might make an exception if TweakWeek 2 still gets to happen (I’m honestly quite unsure if it will), but that a lot that I must think about).

Anyway, I’ll still be around on IRC! I like my friends! You can talk to me!

And as of now, UnthreadedJB’s plan has my full support (don’t ask me details, sorry. No, [s]he’s not making a new jailbreak.).

---

Now, it’s opinion time.

As much as I am against the usual small circle that gets access to jailbreaks for whatever purposes, I find it completely unfair not just to him, but to everyone to leave saurik out of it this time, being that 99.9% of users require Cydia to install stuff, and most of demanded software in the Cydia Store are tweaks, which only works thanks to CydiaSubstrate.

I have no comment regarding the whole China deal. Fuck you, evad3rs. I’m sorry, but fuck you. Making a jailbreak doesn’t compensate for this fact. Charge for a jailbreak, put video ads in it, I don’t give a shit. Anything is better than this and I don’t agree to your fucking point of view. What I’d honestly also like to know is whether if a shady American company with a competitor App Store like the one in China came to you and offered money for you to add it instead of Cydia to the jailbreak for all American users to use by default, if you’d do it. And I’m unsure what answer would be worse.

Goodbye,
Daniel.

Patching iphone-gcc binaries to armv7s

Theos Issue #53 is, as described:

and typing “$THEOS/bin/nic.pl”
my MobileTerminal executes:
Illegal Instruction: 4

So, this means that perl got us an Illegal Instruction error. Oh no!

The solution is put below, which is:

sed -i'' 's/\x00\x30\x93\xe4/\x00\x30\x93\xe5/g;s/\x00\x30\xd3\xe4/\x00\x30\xd3\xe5/g;' /usr/bin/perl
ldid -s /usr/bin/perl

There’s the solution! But once it’s there, why this blog post?

Well, this issue re-surfaced on #theos earlier today, so I figured I’d go around and find out why this is so. I ended up learning a bit more about how ARM Assembly and etc. work, so well, why not put it in a blog post? This tumblr sucks anyway.

So, the second line of the previous script just re-signs the binary, so we can ignore it. The first line applies a binary patch. The question is, what does it patch?

Since I don’t in fact have an iPhone 5, let’s assume he put the Illegal Instruction-crashing program in gdb and discovered the bad instruction by seeing where pc is. As for us, we’ll have to do this by looking at the patch itself.

We shall take the ntpdate binary, which was cited as an example of something the patch made work in the blog post it was initially introduced.

otool -tV ./ntpdate > ./ntpdate_disassembly
# apply binary patch
otool -tV ./ntpdate > ./new_ntpdate_disassembly

diff -u ./ntpdate_disassembly ./new_ntpdate_disassembly

With this diff, we get this change repeatedly:

-000026a0   e4933000    ldr r3, [r3]
+000026a0   e5933000    ldr r3, [r3]

-000027e0   e4d33000    ldrb    r3, [r3]
+000027e0   e5d33000    ldrb    r3, [r3]

So! The issue was at the generated opcode for the ldr and ldrb instructions, and when changing e4 to e5 it magically appears to work! OK!

There are many questions surrounding this still, and after a talk with DHowett about the issue, he pointed out to me that this was in fact a known issue with Telesphoreo’s csu’s crt1.o’s instructions.

Indeed! In the binary, ldr r3, [r3] instructions with the e4 opcode prefix only happened in functions exported by crt1.o (_crt_basename, __start). The same instruction when in other places in the binary has the other (correct) prefix!

So as it turns out, the issue isn’t even an obscure problem with iphone-gcc generating bad instructions that worked on armv6 and armv7 undocumented and not on armv7s, or an obscure instruction which worked on armv6-7 and not on 7s. Rather, it’s an issue with the crt the toolchain used, which for some reason contains in its machine code a non-standard opcode which worked on armv6 and armv7 but not on armv7s.

The task is now to replace it with the proper armv6 opcode.

From that, the approach to find the correct opcode out is, at its simplest:

cat >test.s <<EOF
start:
      ldr r3, [r3]
      ldrb r3, [r3]
EOF

as -arch armv6 test.s -o test.o
otool -tV test.o

And there we have it:

test.o:
(__TEXT,__text) section
start:
00000000    e5933000    ldr r3, [r3]
00000004    e5d33000    ldrb    r3, [r3]

So finally, analyzing the patch closely:

s/\x00\x30\x93\xe4/\x00\x30\x93\xe5/g;s/\x00\x30\xd3\xe4/\x00\x30\xd3\xe5/g;

This, in fact, can be reduced to two substitutions:

• 00 30 93 e4 -> 00 30 93 e5
• 00 30 d3 e4 -> 00 30 d3 e5

Knowing that ARM instructions are formatted to binary back-to-forth (that is, if an opcode is e4 93 30 00, it’ll be written to the binary as 00 30 93 e4)¹, understanding why the patch works is easy.

The big question remains, and can’t be answered: How and why did crt1.o get built with that faulty opcode? We are left with:

[01:10:56] <@DHowett> anyway: [they] realized that [they] never knew where exactly that file came from

Hacking the iOS Spotlight, v2

Introduction

This article is purely technical. If you want to know a bit about the history of SearchLoader, go to this blog post.

All of the data in this article reflects iOS 6. There has been an intermediate number of changes from iOS 5.

History: The TL prefix for everything around SearchLoader and my projects related to Spotlight stands for theiostream spotlight. It looked better than SL, I could certainly not take SP, and TL also looked nicer than TP.

This article limits itself to the basic process of a search, and the creation of Search Bundles and Extended Domains. It does not go into detail on how each component of the Search framework work internally.

There are a couple of things which are irrelevant for this article, but some supposedly interesting areas of this subject might still make new articles. More precisely:

• SPContentIndexer: how does it interact with the Content Index; how can we perform our own searches? (SMS app does that!)

• IPC: How do the search-related process intercommunicate? This should be an easy question to answer, but I was never interested enough to look it up! :o

• Domain registration: How are domains registered out of the Extended domain scope? Regarding the search process itself’s internals, this is the only key point I’ve never closely looked into, and which I suspect isn’t much simple.

---

An overview of the search process

Spotlight is divided between two layers: SpringBoard (UI) and searchd (search).

SpringBoard has four SBSearch... classes:

• SBSearchTableViewCell: The UITableViewCell subclass for the Spotlight table view. It has nothing special regarding the search process. Whole articles could be written on cheating the Search Table View, though.

• SBSearchView: The main view for Spotlight. It has as subviews an UISearchBar, an UITableView and, on the iPad, a SBNoResultsView for stylish purposes. This view also handles some layouting code for the table view.

• SBSearchController: It serves as a bridge between the table view and SBSearchModel. It is the table view and search bar delegate, and transfers information from searched content to your nice-looking results.

• SBSearchModel: A subclass of SPSearchAgent. As a subclass, it handles the timer for search results to vanish after a while, obtaining images for display identifiers and the final launching URL for the result, from data it holds.

SPSearchAgent, in SpringBoard, is incorporated by SBSearchModel’s shared instance.

SBSearchController asks for it to do what it’s meant to do: Take a query string, turn it into a SPSearchQuery and send it to searchd.

The searchd layer uses SPBundleManager to load all existent search bundles (placed at /System/Library/SearchBundles or, with SearchLoader, /Library/SearchLoader/SearchBundles, and then it gets out of them a set of datastore objects.

Datastores on Search Bundles are NSObject<SPSearchDatastore> * objects. These objects, through an API specified by the SPSearchDatastore protocol, perform a search through the query they receive and produce a SPSearchResult.

Search Results are sorted by an integer named a search domain. Each search bundle provides a set of domains it “owns”.

When creating SPSearchResults, internally or externally they will get placed inside a SPSearchResultSection object, which will be assigned to a domain.

Multiple sections can be added under the same domain, as done by Application.searchBundle. The only search bundles to use multiple domains are iPod.searchBundle (due to unknown purposes) and Extended.searchBundle (each index gets one domain).

Back to SpringBoard, it gets sections for each domain and places them in the table view.

Yet, there is some special attention that should be paid to Extended.searchBundle and Spotlight Bundles.

Extended.searchBundle reads from database or database-like entries in some files to generate its results. The following content describes the generation of these entries. But as you can notice, they are database entries. Therefore, this method of displaying search results should be used when results are not generated dynamically, but when they can be indexed.

SearchLoader edits com.apple.search.appindexer.plist (the AppIndexer daemon’s launchd.plist) so it’ll load MobileSubstrate. With this, it manages to control Spotlight Bundle loading.

Every time searchd is initialized (when Spotlight is brought up), it invokes the AppIndexer daemon.

On launch (usually), this daemon finds existing Extended Domains through SPGetExtendedDomains(). This function reads from /System/Library/Spotlight/domains.plist and returns an array of dictionaries. These dictionaries contain this domain’s display identifier (which reflects the generated search results refer to), a category (a string which usually has the format <Name>Search) for it (a way to differentiate different search bundles/extended domains with the same display identifier due to referring to the same app) and required capabilities. This sort of dictionary is, therefore, named extended domain.

Before going further, it’s important to introduce the file hierarchy for Spotlight Bundle databases, etc. Files related to extended domain with display identifier com.yourcompany.test and category TestSearch will be placed at /var/mobile/Library/Spotlight/com.yourcompany.test. The files are:

• TestSearchindex.spotlight: Content index to index search results. Managed by CPRecordStore in AppSupport and the ContentIndex framework;
• TestSearchindex.sqlite: SQLite database managed by CPRecordStore and ContentIndex.framework to index search results;
• updates.TestSearch.spotlight: Content index to track desired updates to the database/content index.

From these extended domains, it uses SPDomainHasUpdatesFile() to determine whether the updates file for an extended domain is empty. In case it is non-existent or contains updates, an AppIndexer instance is initialized with information about this extended domain.

Here, Spotlight Bundles (finally) get in the scene. They are loaded from /System/Library/Spotlight/SearchBundles. Through a principal class of NSObject<SPSpotlightDatastore> * type, it generates a dictionary with specific keys which tells AppIndexer how it should index results into the content index/database, from an identifier. If the updates file is empty, a list of identifiers is created by the spotlight bundle itself. Else, the contents of the updates file are used. Therefore, it can be said that the updates file tracks identifiers that require indexing from the Spotlight bundle.

After getting this data, AppIndexer asks searchd to update the actual database/content index. Meanwhile, one might ask Where do the identifiers for the update file come from?. The only native extended domain, SMSSearch, uses a whole direct ContentIndex wrapper to write to its update file (the IMDSpotlight function family from IMCore). But happily, we don’t have to either use ContentIndex, nor link to IMCore. Apple provides some APIs in SPDaemonConnection, or even a whole framework just about that: Spotlight.framework with SPSpotlightManager.

And then we go back to the top. Extended.searchBundle will use SPContentIndexer to look up contents of existent content indexes/databases and from them build Search Bundle-like results which will go to SpringBoard.

This finishes my Spotlight overview. The further sections will describe, respectively, the structure of a search bundle, an extended domain Spotlight Bundle (documenting libspotlight – a part of SearchLoader –’s APIs), how to make your bundle loadable by SearchLoader, some details about the SearchLoader tweak itself, SearchLoaderPreferences, and then will conclude.

Search Bundles

Search Bundles are composed of a principal class, which is the Search Bundle datastore. It conforms to the SPSearchDatastore protocol.

SearchLoader plugins actually should conform to the TLSearchDatastore protocol, which conforms to SPSearchDatastore and adds one method.

TLSearchDatastore

• - (void)performQuery:(SPSearchQuery *)query withResultsPipe:(SDSearchQuery *)pipe;

In this method, the search bundle should send its generated SPSearchResult or SPSearchResultSection objects back to searchd so it can be shown in the SpringBoard layer.

This method takes as arguments query and pipe. They are the same (as of iOS 6: SDSearchQuery is a subclass of SPSearchQuery), yet theoretically query should be used to obtain information regarding the search query (essentially, its query string, obtainable through the - (NSString *)searchString; method), and pipe to send results back to searchd, through the following methods:

• - (void)appendSection:(SPSearchResultSection *)section toSerializerDomain:(NSUInteger)domain;
• - (void)appendResults:(NSArray *)results toSerializerDomain:(NSUInteger)domain;

In these methods, the domain argument should always be the search domain taken by the search bundle, the section parameter should be an initialized SPSearchResultSection to contain the desired search results, and results should be an array of SPSearchResult objects.

In case there is usage of the below-described -(BOOL)blockDatastoreComplete method and at some point asynchronous behavior happens, you should call -[SDSearchQuery storeCompletedSearch:], passing self as a parameter, and pipe as an object.

• - (NSArray *)searchDomains;

It should return a NSArray object with NSInteger objects as its contents. Each NSInteger should hold an integer to serve as its taken search domain.

Due to a Loader limitation, in SearchLoader-loaded plugins only one search domain should be taken, else unknown results may be yielded.

• - (NSString *)displayIdentifierForDomain:(int)domain;

This method should return a NSString object to represent the display identifier for a given domain. This display identifier is usually the application which search results reflect.

• - (BOOL)blockDatastoreComplete;

To perform some asynchronous-only tasks inside your search bundle or delegate-calling requesters, you can return YES on this method to block the -[SDSearchQuery storeCompletedSearch:] method, therefore not progressing further in the search process and then rendering result committing from the datastore impossible.

Later, a call to -[SDSearchQuery storeCompletedSearch:] should be placed as described above for the result to be actually committed, and obviously, NO should be returned here then for your call not to be subsequently blocked.

libspotlight APIs

The following libspotlight functions can be used for convenience or are required during the development of SearchLoader-loaded search bundles:

• NSUInteger TLDomain(NSString *displayID, NSString *category);

(The internals of this function will be discussed further, and with it the need for a category parameter, which is characteristic of extended domains.)

This function gets the domain for a given display identifier (usually of the application which search results reflect) and a category string (defined above).

This must be the way to obtain the domain for a SearchLoader plugin, to avoid issues with other plugins.

• void TLRequireInternet(BOOL require);

This enables or disables the status bar activity indicator in SpringBoard. This should be used if you are loading content from the Internet.

This function is completely unrelated to the -blockDatastoreComplete method from TLSearchDatastore.

Miscellaneous

• If you, for some reason, cannot use -blockDatastoreComplete to order searchd to wait for asynchronous tasks, you can use CFRunLoopRunInMode() (so you can set timeouts, rather than CFRunLoopRun() where you can’t) to stop it from progressing without results being properly committed. It can later be CFRunLoopStop()ped.

A convention (made by me) states that you should unless extremely required never take over 3 seconds with Internet requests.

Spotlight Bundles

SPSpotlightDatastore

• - (NSDictionary *)contentToIndexForID:(NSString *)anId inCategory:(NSString *)category;

From parameter anId, a string which serves as an *identifier, this method should return a NSDictionary object with specific keys to represent a result. The category parameter is the extended domain’s category.

The following keys can have values assigned for in the returned dictionary. They are all constants defined in the SearchLoader headers, and part of Spotlight.framework:

• kSPContentContentKey: The content of the search result. The query string matching this one is what defines whether this result should or not be displayed.
• kSPContentTitleKey: The title for the search result.
• kSPContentSummaryKey: The summary label of the search result.
• kSPContentSubtitleKey: The subtitle label of the search result.
• kSPContentAuxiliaryTitleKey: (iPad only) The auxiliary title for the search result.
• kSPContentAuxiliarySubtitleKey: (iPad only) The auxiliary subtitle for the search result.
• kSPContentCategoryKey: The category. Use is unknown.
• kSPContentExternalIDKey: The external identifier of the result. Use is unknown.

I should provide an image specifying which labels are which graphically soon. Meanwhile, you’ll have to experiment with it ;)

• - (NSArray *)allIdentifiersInCategory:(NSString *)category;

This should return an array of NSString objects to be passed into -contentToIndexForID:inCategory: as the anId parameter.

This method is called when the content index/database for given category is empty, and therefore it needs all existing data related to it put into identifiers, which will initially populate them.

An identifier has no proper definition nor standard, except the one that if it does not conform to URL standards it will not be put into the default-generated URL for it. The domain will be used instead. It can be as it best fits your parsing needs on -contentToIndexForID:inCategory:. More details regarding default URL generation can be found below on the URL correction InfoBundle plist keys’ documentation.

To set a custom URL for a Spotlight bundle, the TLCorrectURL... InfoBundle keys should be used. More details can be found below. This API is quite limited at the moment, but it can be expanded if a specific request regarding it is placed.

Updates File Manipulation

The following SPSpotlightManager method from Spotlight.framework can be used to modify the Updates file:

• + (id)sharedManager;

Obtains the shared instance for the SPSpotlightManager class.

• - (void)application:(NSString *)displayID modifiedRecordIDs:(NSArray *)identifiers forCategory:(NSString *)category;

This method adds the identifiers, described as NSString objects inside the identifiers parameter, to the updates file of extended domain of display identifier displayID and category category.

Content Index/Database Manipulation

• SPSpotlightManager: - (void)eraseIndexForApplication:(NSString *)displayID category:(NSString *)category;

This method deletes the /var/mobile/Library/Spotlight files for certain category of certain application for given display identifier.

• SPDomainManager: - (void)notifyIndexer;

This method triggers AppIndexer, which will perform its on-launch tasks (update extended domains which require updating).

InfoBundles

InfoBundles are document packages (bundles without executables) placed inside /Library/SearchLoader/Applications/. They tell SearchLoader which search/spotlight bundles placed at their respective directories should be loaded.

Required Keys

• LSTypeIsPackage: Should always be set to true.
• SPDisplayIdentifier: String representing the display identifier of the app which search results refer to.
• SPCategory: Category for the plugin.
• TLDisplayName: Display name for the plugin.

Required Keys for Search Bundles

• TLIsSearchBundle: If set to true, defines that this plugin is a search bundle.

Required Keys for Extended Domains

• SPSearchBundle: The name of the Spotlight bundle related to the extended domain.
• SPDatastoreVersion: This value is mostly unused. Should be set to integer 1.

Optional Keys

• TLCorrectURL: Boolean which defines whether SearchLoader should attempt to correct the URL generated by a search bundle/extended domain. Since you can generate your own URLs with search bundles, this is only intended to be used with extended domains.

The below keys show the process of creating your corrected URL with InfoBundle keys. It should be noted that the generated string should be a valid URL, else it will have no effect.

Correction works based on the manipulation of the original URL string. On search bundles, they are custom, and on Spotlight bundles, they take the following format: search://displayID/category/identifier.

It shall be noted that if identifier does not conform to URL standards, the original output URL after processing this string will have the search://domain/record-entry-ID format.

• TLCorrectURLFormat: String which identifies a format string for the new URL. It should contain:

• Substring <$ID$> expands to the display ID.

• Substring <$C$> expands to the category.
• Substring <$D$> expands to the domain.
• Substring %@ will be replaced by the selection of text from the original result URL defined by the keys below.

If there is no defined format and yet a delimiter, the default format search://<$ID$>/<$C$>/%@ will be used.

• TLCorrectURLStartZero: Boolean which defines whether the selection from the original URL’s text starts at the beginning. Takes precedence over TLCorrectURLStartDelimiter.

• TLCorrectURLEndLength: Boolean which defined whether the selection from the original URL’s text ends at the string’s end. Takes precedence over TLCorrectURLEndDelimiter.

• TLCorrectURLStartDelimiter: Required if TLCorrectURLStartZero is absent. Defines the delimiter string for the start of the selection.

• TLCorrectURLEndDelimiter: Required if TLCorrectURLEndLength is absent. Defines delimiter string for the end of the selection.

Optional Keys for Extended Domains

• TLQueryLengthMinimum: Integer which represents the minimum character count for the content index/database for this extended domain to be searched.

SearchLoader

In the SpringBoard layer, SearchLoader changes:

• Emptying the _prefixWithNoResults instance variable every time -[SPSearchAgent setQueryString:] is called, therefore making every query valid, as opposed to queries with only valid prefixes. This logic works with cases such as “if there’s no Nol contact, there’ll be no Nolan, but doesn’t work with, for instance, Calculator, in which 1- is valid and 1-1 is not.

• Hooking -[SBSearchModel _imageForDomain:andDisplayID:], allowing search results for non-existent apps to exist and still have icons in the table view. For instance, YouTube Search doesn’t require the YouTube app, yet should have an icon.

• It hooks -[SBSearchModel launchingURLForResult:withDisplayIdentifier: andSection:], to apply the changes asked for by the TLCorrectURL... InfoBundle keys.

In the searchd layer, the core hooks are:

• SPGetExtendedDomains(): Every SearchLoader plugin is faked as an existing extended domain, even being a search bundle. This provides a healthy domain for our search bundles and lets our spotlight bundles to be loaded.

• -[SPExtendedDatastore searchDomains]: This prevents our search bundles’ domains to be registered by Extended.searchBundle. This is essential, else an exception will be thrown due to two search bundles having the same domain – our plugin and Extended. With this put aside, our bundle is loaded without any major complication by SPBundleManager.

These are the other hooks:

• SPDisplayNameForExtendedDomain(int): This applies the chosen display name on the InfoBundle.

-[SPContentIndexer beginSearch:]: This is used to apply the restriction from the TLQueryLengthMinimum InfoBundle key.

• NSBundle/NSFileManager Path Hooks: Hooked to allow bundles at custom locations (in this case, /Library/SearchLoader/SearchBundles) to be loaded.

-[SDSearchQuery storeCompletedSearch:]: Hooked to allow TLSearchDatastore’s -blockDatastoreComplete method to be implemented.

• -[SDClient removeActiveQuery:]: Avoids a crash previously caused by a cached value which states SearchLoader was loading from Extended.searchBundle after finishing a query.

On AppIndexer, domain hooks are placed and the following:

• SBSCopyBundlePathForDisplayIdentifier(NSString *): This tricks AppIndexer into getting SPSearchBundle and SPDatastoreVersion keys from our InfoBundle, not the actual app bundle. This is required to avoid file patching and allow extended domains for apps which are not installed.

Lastly, for the TLRequireInternet(BOOL) function from libspotlight to work, a small Darwin notification system is placed inside the SpringBoard layer of the tweak, and when it receives a notification, it accordingly changes whether the status bar activity indicator is or not activated.

SearchLoaderPreferences

SearchLoader also hooks into Preferences to allow the native Spotlight preferences to know about our own plugins. It only applies the core hooks when SearchSettings.bundle is loaded.

Yet, it also adds a preference bundle of its own which thanks to rpetrich and DHowett’s libprefs, can load – just like PreferenceLoader – preference entry plists! So you are allowed to – much like PreferenceLoader – place your plists exactly as you would with PL on /Library/SearchLoader/Preferences. Neat, huh? :)

SearchLoader Limitations/Bugs

• It does not allow multiple domains for search bundles, and unknown consequences may happen if a search bundle attempts to do so. There is, though, no known reason for this to be allowed.

• While search bundles can be placed in /Library/SearchLoader/SearchBundles, Spotlight bundles require to be placed at /System/Library/Spotlight/SearchBundles. The reason for this is purely laziness.

• SearchLoader creates empty Content Indexes for every search bundle, generating a number of empty-and-unused indexes at ~/Library/Spotlight.

---

Conclusion

After 8 months of work in this area and some hours in this blog post (naturally not only on Loader, I’ve made my own share of Loader plugins to be released alongside it), I present this research. I hope it turns out to be useful. Seeing cool things being done with this is the best thing I could ever hope to achieve by making this.

I’d like to thank AAouiz and cydevlop for coordinating the Spotlight+ and SearchResults projects, which drove Loader to be created, and in no particular order Maximus, Cykey, DHowett, rms, fr0st, Nolan, ac3xx, his delightful wife, cj, Optimo, saurik and so many others who helped (directly or indirectly) to make this possible.

Finally, Loader has some fails, as the above section states, but I gladly take feature requests or bug reports.

About me:
I’m Daniel Ferreira (@ferreiradaniel2), typically called theiostream around the jailbreak community, author of some jailbreak tweaks, and other stuff. For more, http://theiostream.com. ;)

SearchLoader

It’s been 7 months since I published my first article on this subject: Spotlight. Yes. That long. I will soon (that is, in a matter of hours) be writing a new technical writeup on the subject, named Hacking the iOS Spotlight, v2.

Let me start by telling you a bit about how research was prior, and after the publishing of the article.

The whole thing started with a project I had with Ariel Aouizerate and Eliran Manzeli nicknamed Spotlight+, which aimed to extend Spotlight by displaying selected search results in a different way. This has not yet been implemented, even though it was the start of it all, and since the future of this project is yet unknown, I won’t disclose much else about it.

We soon realized that Spotlight lacked the categories of results we all wanted. Ariel thought we should display whatever we displayed through a SearchCommands-like thing, but I figured, since we were doing this, we were going to do it right.

So the project changed its focus to a whole new direction. It was no longer about making Spotlight more practical, but about adding a whole new set of functionalities to it. Adding custom search results.

Then research started. I spent a week studying the SpringBoard, searchd and AppIndexer layers of it, and in a hurry released Hacking the iOS Spotlight in this blog.

The article covered an overview of existing software which extended Spotlight, a bit about the two kinds of search plugins – Search Bundles and the nicknamed Spotlight Bundles, and how both functioned, and a small naïve description of what domains were.

That writeup lacked two vital pieces of information, and had one error at its first release (which was then corrected with further research). It lacked:

• Information about loading Search.framework;
• Information about SPSpotlightManager, which should be used instead of SPDaemonConnection directly for AppIndexer-related purposes;
• The error: The actual purpose of -allIdentifiersInCategory: in a Spotlight bundle.

Ever since, research has progressed into a complete piece of software, named SearchLoader, which, as the name says, loads Search bundles and tricks Extended.searchBundle into loading our Spotlight bundles.

It was no simple task, but it will soon, as one would expect, be opensourced, with a through API documentation (in the upcoming technical article).

Loader started simply as a buggy loading thing, but it soon evolved into a stable Loader + Library set, with a lot of stuff one might need when developing their own search plugins. And when I see the first SearchLoader’s working code (which I had copied in my Mac) and compare it with the current code, I can barely believe it evolved so much.

With some short breaks from this project, I worked on some unfinished tweaks, did TweetAmplius, fixed stuff for iOS 6, wrote some of the Theos Docs, did some work on Oligos, re-beat the Mass Effect saga, played some Minecraft and Civilization, but now, it’s finished! I honestly can’t believe this!

To end this “say everything yet nothing” article, this wouldn’t have been possible without the tools of the trade: class-dump, IDA, dyld_decache, and so forth.

And a huge thanks goes to my friends at the jailbreak community. Without them I would have most likely given up on this ;P.

So, expect cool stuff I’ve coded to come alongside Loader, and I’ll hopefully expect from you some cool plugins for SearchLoader. :)