We’re excited to be working with Portland State University student, @mikayla.c.maki and professor Bart Massey, her advisor, on a project to implement a reference blog application in Goblins based on the example in the Heart of Spritely white paper.
This thread would be a great place to talk about the development of project “CapBlog”.
Mikayla, you’ve already introduced yourself - but please talk a little bit more about the project…
Yay! Obviously @mikayla.c.maki should introduce herself, but I will share the following: @mikayla.c.maki has really impressed me by just how fast she’s been able to consume, critically analyze, and engage with ocap history and lore. It’s really wonderful that we’re able to work with her!
Thanks for the introduction @frandallfarmer and @cwebber! I’ve been at the foresight workshop for the last 2 days but am now shifting gears to focus on CapBlog 
.
My goal with CapBlog is to explore the UI/UX space for OCaps on a multi-user website. I decided to start with a blog because most of the ‘business logic’ is already written in the Heart Of Spritely paper and blogs are simple enough that confirming correctness is conceptually trivial. Some specific questions I’d like to answer in the next three months are:
If all goes well, I’ll also try to create and record a live demo of the interactions described in the paper, with actual packets going over the network!
As work starts happening, I’ll try to post regular progress updates to this thread as well 
That would be great! Woo, I’m excited!
Status update: I spent about 3 hours today, working through the racket tutorial with Christine.
Goblins is a really interesting concept for a framework. All the data storage is done in immutable closures, organized with a map from object identifiers to these closures. Every time an object needs to update it’s state it simply produces a new closure, containing the new state. This makes transactions conceptually simple, just keep the old closures around and restore the mappings. I also got to play with promises a little bit, pipelining seems cool but I’m still a bit fuzzy on how it actually works. I’m really excited to be digging into this project
It was a great call! You’re on a good track and are clearly getting the ideas!
Finally hit my stride with living in SF, this project, work, etc. feeling really good! Spent an hour or so with Christine today and fleshed out the first section of the Heart of Spritely paper. It’s been a process, learning Racket, Emacs, Goblins, and OCaps at the same time is a lot! . Hopefully by the end of this week, I’ll have finished all these tutorials and have a better idea of how all these pieces can be assembled. But I feel really proud that it’s starting to come together . Here’s a screenshot of the running code:
Re: wisp and the HoS paper, I’ve found that it’s raised the complexity of actually implementing these ideas. I’m coding in Racket, the tutorial is in wisp, and the auto-generated translations are in Guile, three different dialects of a still unfamiliar programming language 
. That said, I think that’s an acceptable tradeoff. Reading the HoS paper and seeing the blog’s composability is what inspired me to work with OCaps in the first place! wisp really is ‘pleasantly like pseudocode’ and it would have been harder to get the big ideas in the paper with a paren-heavy dialect of lisp.
Today, I finished the Heart of Spritely paper with Christine
It took a while, but I’m glad we methodically transcribed and understood every code snippet from the goblins tutorial in the racket docs and the HoS paper. I had a bunch of ‘Aha!’ moments today that felt like the culmination of all the learning I’ve been doing for the last month:
Transactions force you to be disciplined about side effects. I attempted to re-write the ^logger example in the paper to record failed accesses along with a flag indicating whether the access was revoked. But the paper’s example used an (error ...) call, which caused the transaction to be rolled back, causing the log to not be updated! We fixed this by switching out the (error ...) for a 'denied symbol. But this could still fail to log if the underlying object throws an error. If we wanted to catch any access Robert does, no matter their result, we have to use an eventual send operation to ensure that the logging ‘commits’ before the procedure is executed.
When I first joined this idea space I thought the Vat metaphor was arbitrary, only useful as a performance optimization and otherwise mucking up the clean ‘it’s all actors and capabilities!’. But with Goblin’s transactions I’ve come to understand it better as the ‘Transaction Boundary’, the $ synchronous operator as ‘Continue this transaction’, and the <- eventual send operator as ‘commit this transaction’. Anything outside the transaction boundary can only be interacted with outside of a transaction, and thus with the <- operator. Understanding this nicely decouples Vats from machines, and gives me ideas for how to blur the boundaries of a vat. As long as the goblins transaction is maintained, why not use the $ to call into an external SQLite database? The possibilities are limitless
Composition enables attenuation, and attenuation enables composition in a really fascinating feedback loop. The reason I’m here and doing this project at all was the ‘guest post with review’ example, tucked away at the bottom of the HoS paper. Implementing this feature shows how these two properties interact to make complex interactions well isolated. In a sense, attenuation increases the composability of a system by simplifying what each interface has to care about. This simplification allows the creation of new roles, e.g. the reviewer, which expand the ways that the system can be interacted with even though attenuation restricts the operations of a capability. It feels like a bit of a contradiction with the way I’m thinking of it, by restricting operations we can create new operations. But it pattern matches well with other ideas like ‘abstraction’ and it feels right in that deep kind of way, so I just need better language for describing this dynamic 
.
And finally: with lisp it’s just S-expressions all the way down :D. This is more of a personal detail, but it’s been nice to unlearn all of these programming constructs that clutter up my intuitive understanding. Really, there are only ever 4 rules:
(...)
That’s it. Nothing else to think about. It’s pleasingly regular and simple In a certain sense, this is the same insight as above (composition through attenuation, attenuation through composition…) but applied at the level of programming languages.
All in all, feeling really excited! Now to actually make all of this Do Things
That was a really great post @mikayla.c.maki! And a great hacking session. Thank you for writing up your observations here, that should make writing the paper easy: you can mine your posts on this thread when constructing the outline of the paper!
Just a historical note here. In E eventual sends are released immediately. In other words, they are not transactional. Tyler Close’s waterken system held messages until the commit, making them part of the transaction.
There is a tradeoff. Delaying messages until the transaction commits adds latency. Back in the days when we were writing to spinning disk, that fact limited how fine-grained we could go. For example, a fine-grained computer chess program ran 1,000 times slower than a parallel Java version. With SSD, that would only be 100 times slower. I estimated I would have had to increase the computation per commit to around 1 ms to make it only 10 times slower. That’s not a big deal for applications that work over the Internet, like Spritely, but it may be for vats running on the same machine.
There is a huge upside in resilience. Waterken provided reliable message delivery, even across crash-restart. A key reason was that holding the messages until commit means that a vat’s most recent commit state is guaranteed to be consistent with all other vats in the system. That’s not true with the E style where finding a consistent state following a crash can be quite hard. It also makes it natural to keep sending a message until you get an acknowledgement for it, which dramatically reduces the number of dropped messages.
It’s fascinating to hear that Spritely’s resilience simply wasn’t computationally feasible for a lot of applications until (relatively) recently. This is a great example of how hardware constraints shape what’s possible. I’ve had a hunch for a little while that non-volatile memory (e.g. https://twizzler.io/) could be really amenable to a goblins-like workload. Will be cool to see what happens!
Edit: Actually, since I last saw it seems twizzler pivoted in a different direction than I was hoping 
. Basically they used to be about persistent pointers that you can hand around a network and store in non-volatile memory easily with some major performance gains. Dunno what’s up with them now 
, seem pretty wedded to ACLs 
Goblins does have a speedier #:reckless? mode that commits directly to the heap instead of doing a transaction. It wouldn’t be hard to extend recklessness into being an option for eventual sends too, should there turn out to be a motivation.
However the only time I ever used #:reckless?, aside in benchmarks, was in trying to see how far I cold push a game prototype. Transactions are just too nice.
You should be able to use #:reckless? mode within a failure domain, such as vats running on a single machine. You just have to make sure all vats commit before releasing a message to the outside.
iiuc it sounds like @alanhkarp is talking about disk transactions, as opposed to what I understand goblins to do, which is just to make each turn of the event loop a transaction, such that heap modifications are rolled back on exceptions – if anything is persisted to disk that’s news to me? I don’t think you should see much of a slowdown in the latter case; it just means sticking messages in a queue as opposed to sending them immediately, no?
Side note, both of the CapTP (Cap’n Proto) implementations I maintain actually do eschew the concept of a vat except insofar as it pertains to the protocol itself, in an effort to fit in more naturally with host language notions of how concurrency is done. But in the case of the Haskell implementation, we still get the benefits of transactions by virtue of using STM to do eventual sends and such (The Go implementation is not so lucky ).
Yes, @isd, it was spinning disk transactions back when I did the work. You may not need that these days, but you’ll need some kind of persistent storage if you want to survive a machine failure.
You don’t need to commit to persistent storage every time, but you must any time you send a message outside your failure domain. You’ll have a hard time recovering to a consistent state if you don’t.
Small update for November 6th, we have connection! Here’s a goofy pic of me and @cwebber making a p2p connection over CapTP, using the example in the racket docs
Next steps: Plan out the live demo of the HoS paper
WE ARE LIVE! Me and @cwebber where able to coordinate the full Heart of Spritely demo, asynchronously, across a half dozen different peer identities, across 2 machines, and a webserver, over the tor network, all with the Racket Spritely Goblins framework
There’s a bunch of discussion and hyper over this on Christine’s mastodon, as well as these two photos of the moment when we realized it was all working
Thoughts:
. At the end of the day, no security model or type system can prevent simple typos, only unit tests!I gotta go decompress now, and think about what all of this means for the future. We’ve got a lot of work to do
Congratulations! And I’m so glad to hear that Goblins’ OCaps crossed the “don’t even think about it” threshold for you.
