Last night, I picked up a book. And when I say a book, I mean two books. One was new and the other old; one's silver and other gold. After nearly a decade, I read Shooting an Elephant again. The essential pathos of someone trapped in a system not of their making, whose identity and behaviour is dictated by unspoken rules, is palpable through out the story. I never got the message with the clarity that I do now - but I was a teenager then.
To see Orwell as merely the author of 1984 and Animal Farm is to do him injustice. Reading through his essays, especially the Lion and the Unicorn throws new light at his ability to communicate adeptly and adequately. Without turning any flowery phrases, he manages to paint a vivid picture of England at war. The part one starts off very directly with "As I write, highly civilized human beings are flying overhead, trying to kill me.", before attacking the essential rationalization of patriotism. But even more cynically, the second part is titled Shopkeepers at War, talking about the private economy's reaction to war (Lend to Defend; Drink Guinness).
I was half-way through "Prevention of Literature", when I switched to the other book. In one of those lapses of judgement (you know the kind), I had bought a copy of Salmon of Doubt. If not for the essays liberally sprinkled throughout the book, I would have regretted that decision. Unfinished books are not meant to be read - for instance, I have read every P.G Wodehouse novel/short-story except for Sunset at Blandings. Hidden in the pages of Salmon of Doubt, between articles for boys' weeklies and random musings about technology is a transcript of an extempore speech.
The speech is titled Is there an Artificial God?. The speech, just like all of his works, is splattered with inconsequential references. For instance, it gets on its way with the following phrase.
I just want to mention one thing, which is completely meaningless, but I am terribly proud of - I was born in Cambridge in 1952 and my initials are D N A!
But it soon settles down into an exploration of man's progress and perhaps for a bit, the lack of it. The ages of sand sequence points out the cornerstones of the last four centuries of human progress - telescopes, microscopes, silicon chips and fibre optics. A quick dip into a quantum universe and back to challenging religious dogmas. To climb out of that argument only to throw in a strawman of money being a completely fictious entity. Eventually concluding that we humans live in a world mainly of our creation - inside our heads or outside.
Reading somehow does not seem to be quite the one-way process most people imagine it to be.
--The possession of a book becomes a substitute for reading it.
-- Anthony Burgess
Brian Shire has put up his slides (835k PDF) of his php|tek talk.
Quite interesting procedures followed to prevent the very obvious cache slam issues by firewalling the apache while restarting it, as well as the priming sub-system they use. Also the cross-server (aka site-vars) seem like a good idea as well - a basic curl POST request moving around json data could potentially serve as a half-reliable cross-server config propogator.
Seeing my code used makes happy ... very happy, indeed.
--I'm willing to make the mistakes if someone else is willing to learn from them.
Most wireless routers come without a DNS server to complement their DHCP servers. The ad-hoc nature of the network, keeps me guessing on what IP address each machine has. Even more annoying was the ssh man-in-the-middle attack warnings which kept popping up right and left. After one prolonged game of Guess which IP I am ?, I had a brainwave.
MAC Address: Each machine has a unique address on the network - namely the hardware MAC Address. The simplest solution to my problem was a simple and easy way to bind a DNS name to a particular MAC address. Even if DHCP hands out a different IP after a reboot, the MAC address remains the same.
Easier said than done: I've been hacking up variants of the twisted.names server, for my other dns hacks. To write a completely custom resolver for twisted was something which turned out to be trivial once I figured out how (most things are), except the dog ate all the documentation from the looks of it.
class CustomResolver(common.ResolverBase):
def _lookup(self, name, cls, type, timeout):
print "resolve(%s)" % name
return defer.succeed([
(dns.RRHeader(name, dns.A, dns.IN, 600,
dns.Record_A("10.0.0.9", 600)),), (), ()
])
Deferred Abstraction: The defer model of asynchronous execution is pretty interesting. A quick read through of the twisted deferred documentation explains exactly why it came into being and how it works. It compresses callback based design patterns into a neat, clean object which can then be passed around in lieu of a result.
But what is more interesting is how the defer pattern has been converted into a behind-the-scenes decorator. The following code has a synchronous function converted into an async defer.
from twisted.internet.threads import deferToThread
deferred = deferToThread.__get__
@deferred
def syncFunction():
return "Hi !";
The value a returned from the function is a Deferred object which can then have callbacks or errbacks attached to it. This simplifies using the pattern as well as near complete ignorance of how the threaded worker/pool works in the background.
53: But before I even got to running a server, I ran into my second practical problem. A DNS server has to listen at port 53. I run an instance of pdnsd which combines all the various dns sources I use and caches it locally. The server I was writing obviously couldn't replace it, but would have to listen in port 53 to work.
127.0.0.1/24: Very soon I discovered that the two servers can listen at port 53 on the same machine. There are 255 different IPs available to every machine - 127.0.0.2 is the same as localhost, but the different IP means that pdnsd listening on 127.0.0.1:53 does not conflict with this. Having reconfigured the two servers to play nice with each other, the really hard problem was still left.
RARP: The correct protocol for converting MAC addresses into IP addresses is called RARP. But it is obsolete and most modern OSes do not respond to RARP requests. One of the other solutions was to put a broadcast ping with the wanted MAC address. Only the target machine will recieve the packet and respond. Unfortunately, even that does not work with modern linux machines which ignore broadcast pings.
ARPing: The only fool-proof way of actually locating a machine is using ARP requests. This is required for the subnet to function and therefore does work very well. But the ARP scan is a scatter scan which actually sends out packets to all IPs in the subnet. The real question then was to do it within the limitations of python.
import scapy: Let me introduce scapy. Scapy is an unbelievable peice of code which makes it possible to assemble Layer 2 and Layer 3 packets in python. It is truly a toolkit for the network researcher to generate, analyze and handle packets from all layers of the protocol stack. For example, here's how I build an ICMP packet.
eth = Ether(dst='ff:ff:ff:ff:ff:ff') ip = IP(dst='10.0.0.9/24') icmp = ICMP() pkt = eth/ip/icmp (ans,unans)=srp(pkt)
The above code very simply sends out an ICMP ping packet to every host on the network (10.0.0.*) and waits for answers. The corresponding C framework code required to do something similar would run into a couple of hundred lines. Scapy is truly amazing.
Defer cache: The problem with flooding a network with ARP packets for every dns request is that it simply is a bad idea. The defer mechanism gives an amazing way to slipstream multiple DNS requests for the same host into the first MAC address lookup. Using a class based decorator ensures that I can hook in the cache with yet another decorator. The base code for the decorator implementation itself is stolen from the twisted mailing list.
Nested Lambdas: But before the decorator code itself, here's some really hairy python code which allows decorators to have named arguments. Basically using a lambda as a closure, inside another lambda, allowing for some really funky syntax for the decorator (yeah, that's chained too).
cached = lambda **kwargs: lambda *args, **kwarg2: \
((kwarg2.update(kwargs)), DeferCache(*args, **(kwarg2)))[1]
@cached(cachefor=420)
@deferred
def lookupMAC(name, mac):
...
The initial lambda (cached) accepts the **kwargs given (cachefor=420) which is then merged into the keyword arguments to the second lambda's args eventually passing it to the DeferCache constructor. It is insane, I know - but it is more commonly known as the curry pattern for python. But I've gotten a liking for lambdas ever since I've started using map/reduce/filter combinations to fake algorithm parallelization.
After assembling all the peices I got the following dnsmac.py. It requires root to run it (port 53 is privileged) and a simple configuration file in python. Fill in the MAC addresses of the hosts which need to be mapped and edit the interface for wired or wireless networks.
hosts = {
'epsilon': '00:19:d2:ff:ff:ff'
'sirius' : '00:16:d4:ff:ff:ff'
}
iface = "eth1"
server_address = "127.0.0.2"
ttl = 600
But it does not quite work yet. Scapy uses an asynchronous select() call which does not handle EINTR errors. The workaround is easy and I've submitted a patch to the upstream dev. With that patch merged into the scapy.py and python-ipy, the dns server finally works based on MAC address. I've probably taken more time to write the script and this blog entry than I'll ever spend looking for the correct IP manually.
But that is hardly the point 
.
What's in a name? that which we call a rose
By any other name would smell as sweet;
-- Shakespeare, "Romeo and Juliet"
Previously while talking about inclusion checks I had included a few helpful digraphs of php includes. Those were drawn with the help of a gdb macro and a bit of sed/awk. But that becomes a real hassle to actually use very quickly while inspecting fairly large php applications.
The solution to repeatable include graphs from php came from the way the include_once override hack was implemented. By overriding the INCLUDE_OR_EVAL opcode in Zend, I could insert something which could potentially log all the various calls being made to it.
That is exactly what inclued does. The extension provides a single function which provides a copy of the collected data, aptly named inclued_get_data().
<?php
include("x.php");
print_r(inclued_get_data());
The above peice of code returns an array with too much information.
Array
(
[includes] => Array
(
[0] => Array
(
[operation] => include
[op_type] => 2
[filename] => x.php
[opened_path] => /tmp/x.php
[fromfile] => /tmp/z.php
[fromline] => 2
)
)
)
Overriding the opcode implies that more information can be collected, like whether the include parameter was a constant (i.e APC can optimize it to a full path), the function the include was in (__autoload *cough*) and the stream wrapper (phar: for instance). The information also includes info about whether the include_once was ignored because of duplication.
Only Data: The extension however does not make any judgements on the code. I have resisted the temptation to put an error log saying "If eval() is the answer, you're almost certainly asking the wrong question" . But there are interesting ways to massage this data into something useful. For example, I just put the following lines into the Wordpress index.php.
$fp = fopen("/tmp/wp.json", "w");
fwrite($fp, json_encode(inclued_get_data());
Loading the JSON data and pushing out a graphviz layout was almost too easy. Here's a clipped section of how my graph ended up looking.
The graph shows about 40-odd includes being hit for a single request, but isn't messy at all. Hopefully, someone finds an actual peice of sphagetti includes which shows off the beauty of this ext (remember, this shows include_once misses & hits).
Hope this helps people debug "Where is this include coming from ?" question I've run into so many times :)
--It was the kind of mental picture you tried to forget. Unsuccessfully.
-- Terry Pratchett, "The Light Fantastic"
Every year, I dread this day. A day when I take stock of my achievements, attempts and failures. But I guess this year too is different.
So many things happened last year. I did the right thing for some people - put family before my career, principles ahead of money, comfort of others ahead of mine. The priorities did change, but I still did some of the things I wanted - travelled a bit, read a pile of books and just hung out with friends for most of the time. Some things caused me to sit up and evaluate my life - they weren't nice things, but I'm better for having weathered them. In short, it does feel good to be twenty five !
And I got some interesting advice today: "Act your age. Drop the 'been there, done that' attitude and stop acting like you're 35".
I will, oh ... I will.
--Old age is too high a price to pay for maturity.
I wouldn't have paid to watch this movie. Instead of the dark, deep conflict between power and losing control, the movie is peppered with a self-obsessed Peter Parker and a depressed MJ. Neither of which are wrong or bad, but the emotional drama is also not used to any effect.
No Villains: Topher Grace is the only saving grace in the whole movie. His transformation into Venom gives a decent bad guy. The sandman puts me to sleep and Harry flips-flops in & out of villainhood. The small tugs & pulls of conflict in the former was brought out in the wrong light - less of dark desperation than I'd imagine.
The scenes looked peiced together from clips inspired from movies. The comedy just didn't fit in and neither did the cool Peter Parker doing a Saturday Night Fever sequence down the boulevard. And the tower spire with the dark spiderman on top looked nearly identical to the scenes from Underworld, including the jump off the top. It attempts to do a fair bit of moralizing, but the message falls far short of the original With great power comes great responsibility, even after they got Stan Lee to say it ("I guess one *Superhero* can make a difference.", 'nuff said). But there is one quote off the movie which I'm still going to take to heart.
Aunt May: You start by doing the hardest thing: you forgive yourself.
All in all, I think I forgive myself for wasting those hours of my life.
--Children begin by loving their parents.
After a time they judge them.
Rarely, if ever, do they forgive them.
-- Oscar Wilde
APC user cache is cool. It provides an easy way to cache data, in the convenient form of hashes & more hashes within them, to share the data across processes. But it does lend itself to some abuses of shared memory which will leave your pager batteries dead, disk full of cores and your users unhappy. Eventually, the blame trickles down to splatter onto APC land. Maybe people using it didn't quite understand how the system works - but this blog entry is me washing my hands clean of this particular eff-up.
apc_fetch/_store: The source of the problem is how apc_fetch and apc_store are used in combination. For example, take a look at countries.inc from php.net. The simplified version of code looks like this.
if(!($data = apc_fetch('data'))) {
$data = array( .... );
apc_store('data', $data);
}
There is something slightly wrong with the above code, but the window for the race condition is too small to be even relevant. But all that changes the moment a TTL is associated with the user cache entry. For a system under heavy load, all hell breaks loose when a user cache entry expires due to TTL. Let me explain that with some pretty pictures and furious hand-waving.
Each of the horizontal lines represent an individual apache process. The whole chain reaction is kicked off by a cache entry disappearing off the user cache. Every single process which hits the apc_fetch line (as above), now falls back into the corresponding apc_store. The apc_store operation does not lock the cache while copying data into shared memory. So all processes are actually allowed to proceed with the copy into shared memory (yellow block) in parallel. The actual insertion into the cache, however is locked. The lock is hit nearly simultaneously by all processes and sort of cascades into blocking the next process waiting on the lock.
Lock, lock, b0rk !: The cascade effect of waiting on the same lock eventually results in one process locking for so long that it hits the PHP execution timeout Or the user get bored and just presses disconnect from the browser. In apache prefork land, these generate a SIGPROF or SIGPIPE respectively. If for some reason that happens to be while code inside the locked section is being executed, apache might kill PHP before the corresponding unlock is called. And that's when it all goes south into a lockup.
So, when I ran into this for the first time, I did what every engineer should - damage limitation. The signals were by-passed by installing dummy signal handlers and deferring the signals while in locked sections. Somebody needs to rewrite that completely clean-room, before it is going to show up in the pecl CVS. The corresponding cache slam in the opcode cache is controlled by checking for cache busy and falling back to zend_compile - but the user cache has no such fallbacks.
I wish that was the only thing that was wrong with this. But I was slightly misleading when I said the copy into shared memory was parallel. The shared memory allocator still has locks and the actual allocation looks somewhat like this for 3 processes.
The allocations are interleaved both in time and in actual layout in memory (the bottom bar). So adjacent blocks actually belong to different processes, which is not exactly a very bad thing in particular. But as the previous picture illustrates, every single apc_store() call removes the previous entry and free's the space it occupies. Assuming there are only three processes, the free operation happens as follows.
The process results in very heavy fragmentation, due to the large amount of overlap between the shared memory copy (apc_cache_store_zval) across processes. The problem neatly dove-tails into the previous one as the allocate & deallocate cost/time increases with fragmentation. Sooner or later, your apache, php and everything just gives up and starts dying.
There are a couple of solutions to this. Since APC 3.0.12, there is a new function apc_add which reduces the window for race conditions - after the first successful insertion of the entry, the execution time of the locked section is significantly reduced. But this still does not fix the allocation churn that happens before entering the locked section. The only safe solution is to never call an apc_store() from a user request context. A cron-job which hits a local URL to refresh cache data out-of-band is perfectly safe from such race conditions and memory churn associated.
But who's going to do all that ?
--In the Beginning there was nothing, which exploded.
-- Terry Pratchett, Lords and Ladies
The great and awesome teemus has come up with a theme song for APC. Set to the background score of Akon's Smack That !, it goes something like this.
Cache that, give me some more, Cache that, don't dump some core, Cache that, don't hit the _store, Cache that, ~ oohoooh ~
In a sort of related note, I might be singing this somewhere.
--Everything in nature is lyrical in its ideal essence, tragic in its fate, and comic in its existence.
-- George Santayana
Man's unique agony as a species consists in his perpetual
conflict between the desire to stand out and the need to
blend in.
-- Sydney J. Harris
It is a trade-off. Everybody needs their own personal rebellion and to be welcomed back into the herd - like the prodigal son of the fables.
Some people get stuck half-way. Some people don't even start.
--Not to be a socialist at twenty is proof of want of heart;
to be one at thirty is proof of want of head.
-- Georges Clemenceau
posted at: 04:05 | path: /philosophy | permalink |
Apparently the median for my office persona is a straight faced automaton. But about a fortnight ago, all that changed. I realized the obvious solution in one of those moments of lucidity - "don't worry". It might sound simple, but it takes a lot of effort to side-swipe the society's influence to actually say don't sweat the small stuff to yourself. And it takes great tragedy too, to separate the small things from the large and the large from the inconceivable. So here's how the book would look like if I wrote a book about it - which I won't, but I still love the cover (click image for bigger version).
I've generally stopped being upset by small setbacks. Is only life, as they say it. But it took a quite large learning curve before I dropped into this chilled out world, where everything's cool (eventhough it is summer) and I get the important things done right. I guess I'm not the first one to actually go down this path. Rather than say it out myself, let me pull out those words from the pen of Bill Watterson and the mouth of that cheeky six-year-old.
But could that really be ? I suppose so ... it works at this end.
--"That depends a good deal on where you want to get to," said the Cat.
"I don't care much where--" said Alice.
"Then it doesn't matter which way you go," said the Cheshire Cat.