Please Review: The Heart of Spritely (HoS)

Architecture Whitepapers
, , ,
21

I’m on page 20, and I haven’t seen anything about chained-attenuated delegation. I think this topic is important enough (and different enough from ACL systems) to warrant discussion long before this point.

1 Like
22

Finished sections 1-3. My comments are in the Google Doc.

2 Likes
23

Thanks for the very rapid turn-around!

1 Like
24

I aim to please, which is easier to do now that I’m unemployed. :grinning:

2 Likes
25

Mirroring Randy’s thanks on the rapid turnaround! This is great feedback.

I know you’ve got other “this is unfamiliar” comments about the syntax I’m sure, but notably though the import to google docs lost something important: it switched all the code examples to variable width. Not a great fit for the indentation-based syntax we’re using…

26

The font change had nothing to do with the problems I had with the code examples. I had the same issues when reading the HTML version.

27

Hello :wave:

I’ve gone through the document and have some comments which I’ve left below. I’ve done them based on an ODT document which I exported myself so I apologise if some of my page numbers don’t align with the ones in the document linked in the forum. Overall I want to say that I really liked the document and thought that the examples (specifically the blog example) was really good at demonstrating Goblins’ tooling and techniques. I think that from " 3.2.5. Guest post with review" until the end is good but it could do with more examples.

I thought I’d mention my experience with scheme. I did find before I started working on Spritely a similar problem with understanding scheme (I usually could get the main ideas if I tried, but it wasn’t always straight forward). I would say though, I thought I’d have a much harder time getting used to scheme when I began working on Spritely and basically after a day or so of looking at scheme code and working with it, it was as easy as any Python, JS, etc. I’m not too sure if I just had an unusually easy time with it but I think scheme feels more difficult and scary than it deserves to.

My feedback:

  • Page 2
    • not explaining how or way solitaire gets access to the keyboard & mouse input.
  • Page 7
    • How come it’s using this unknown mcell in the ^cgreeter example? (^cell) would make more sense)
    • You could rename our-cgreeter to julius to match with the example in 3.1.4 (Asynchronous message passing)
    • Doesn’t it in the future print out " Heard back: Hello Lear, my name is Julius!"?
  • Page 9 & 10
    • Could use a bit of explaination I think something along the lines of:

While we’ve seen you can use on to resolve promises and see the result, you can also use them as if they’ve already resolved. In the following example we’re creating a car factory for the car manufacturer “Fork” and with this creating a blue “Explorist”. We can then use the car we made. You can see we’ve done it in two ways. The nested version where you resolve each promise first with on, getting the result and then send messages and one where you send messages to the promise itself, as if it was the object we hope it’ll resolve to.

A side note, conventionally in Goblins code we call promises “vows”. You can hopefully see the second is a lot easier to read and think about. It also has other advantages such as if this car lives on another computer, all these messages are sent and can be executed at once instead of having many round trips.

NB: This will want re-writing or refining, it’s a very rough draft.

  • Page 11 (and kinda page 14):

    • It took me (someone familiar with scheme, albeit not guile) multiple reasons to realise that cons was basically prepend I wonder if there’s a more obvious way of writing this?
    • I wonder if we should specifically mention that cons is prepending to the list, this could just be a comment when using it.

  • Page 12:

    • the update method does not use a named let, it’s making the ^editor become a new version of itself with the new information.

  • Page 13:

    • Bit confused what’s happening with Roberts post. Is Robert drafting a post and sending it OOB to Lauren who is then making this or is the text: “Run by Lauren:” incorrect and this is in fact run by Robert?

  • Page 16:

    • Not sure why but, I found the sealers implementation a bit difficult to understand. I think it is guile’s record stuff being less than obvious when you initially look at it.
    • Something that’d be useful would maybe have the explanation of sealers/unsealers/brands with the usage of them and then either below or in the appendix have the implementation.

  • Page 22:

    • The append to log message could be using <-np. I think this is probably intentional as I don’t think you’ve introduced <-np so maybe it’s good as it is.
    • The explanation about Robert potentially giving admin-for-robert to someone else, it sounds like you’re only going to hold Robert accountable for actions he himself did (not someone who he’s passed it on to). Maybe it should be something like “Lauren will hold Robert accountable for any actions done with the admin-for-robert capability”.

  • Page 25:

    • I think the explanation of promise pipelining would work better on page 9 & 10 where you introduce it.
2 Likes
28

Hello!

I’ve read the HoS document up to Section 3 included; it was a pleasant and insightful read! I already know Scheme and the W7 paper, had already read and listened to @cwebber about this topic; after reading this, I have a better understanding of how I as a programmer would use Goblins, and how it would help (I’ve also been doing concurrent programming with Fibers over the last few days, so some parts resonate with me.)

Everything up to Section 3.2 included reads very well for me; the running example of a blog works very well, and it’s great to see features adding up.

Time traveling remains mind-blowing to me. :slight_smile: One question that came to mind is how do you deal with the ever-expanding object graph? Is there a mechanism in place to delete “old object generations”? Minor cosmetic suggestion to make it a tiny bit less intimidating: writing become instead of _bcom (even if it’s unused).

Regarding error handling, I felt like #:catch and #:finally were treated too quickly, and I was wondering how well this would be applicable to more complex programs. Would programs end up being covered with exception handlers? What about transient networking error, when two objects live on different machines? How would I write a program that handles them gracefully, reporting errors to the user and/or retrying? Perhaps this is best answered elsewhere, I don’t know.

Section 3.1.6 on promise pipelining needs some love. :slight_smile: I can see where it’s getting at, and I think performance is a crucial point that should be clearly answered, maybe in Section 3.3. Namely, the strength of message-passing models (and RPCs, etc.) is that they provide network transparency; their weakness is that they invite users use RPCs (and similar) as if they were local function calls, when in fact they have much higher latency. Promise pipelining can help address that, provided developers pay attention to it. The performance pitfalls and solutions need to be addressed clearly IMO.

I found Sections 3.3 onward to be more difficult to read: they’re more dense and have fewer examples. At first glance I’m a bit skeptical about automatic serialization, but I’m not sure whether my concerns are valid here. My first concern is the risk of compatibility mismatches between the serialized form and their live counterpart (I think one of the lessons of orthogonally persistent OSes like L3 was that orthogonal persistence actually complicates things that we’re used to with explicit serialization code, such as being able to read an old version and automatically “upgrade” it to its new live version.) The second concern is the risk of introducing vulnerabilities: if, concretely, the serialized form is store in a file on disk, I can manually fiddle with it, grant authorities to some objects, etc. What about serializing references to objects that live in a remote vat? These are all tricky questions.

Section 3.7 touches on the idea of object identity in a distributed context. I did not understand the unum pattern; maybe a more “hands-on” example might help. I vaguely remember discussions on how to check object identity in the Hurd, given two Mach ports (capabilities to remote objects), which I think involved a mediating server; I could look for it if there’s interest.

That’s about all I have to say for now; I hope it’s useful. Thanks for the refreshing and insightful read!

2 Likes
29

Great! And yes, Goblins on Guile actually uses Fibers under the hood to run vats. (Well, that’s one way to run them, and the main way!)

Oh that’s great to hear!

Well as the primary initial author of a famous time-traveling gnu/linux distribution (Guix, for those who might not know), I am entertained by this.

Goblins is really “quasi-functional”; the most interesting part is how it hides the functional structures from the user allowing the user to focus on time and object relationships without having to deal with monadic plumbing directly.

By default, old object graphs are not persisted, but they can be chosen to be persisted (including for debugging purposes). Transactions are committed to weak hashmaps, so eventually references are collected. (However this is one place where the Guile version is slightly behind Racket’s: the lack of ephemerons in Guile means that objects which reference themselves cannot be automatically collected. Hopefully fixed soon.) I think this would be best understood with use though.

Heh, this has come up multiple times. However, bcom is kind of a Goblins-ism and every time you see it it’s a hint as to this very specific idea (aside from saving two characters every time you type it). It might be that the underscore is a distraction. But it’s just convention, so I’m open to changing it if it would help…

Noted.

Well, one advantage of Goblins’ transactional design is that failures typically get handled “the right way”, so you need exception handlers in fewer places than you might think; the need to write cleanup code is rare compared to most systems. But I should explain this.

It’s a good question, and not yet answered. The answer comes from MarkM’s dissertation, see Chapter 17, “Partial Failure”. Basically an OCapN/CapTP session can be severed, but you can subscribe to find out if this will ever happen in advance by setting up on-severed (supported in the Racket version but not yet ported to the Guile version), which is like a special kind of on that tells you if or when a session has been severed allowing you to reconnect. The other key detail not described in the document yet (it will be, I just haven’t gotten there, it’s coming in the OCapN section) is on “sturdyrefs”, which are like dehydrated references which must be rehydrated (“enlivened”), not unlike how visiting an HTTP URI enlivens an HTTP connection (even if that is typically more brief). All the references shown so far are “liverefs”, basically even over the wire are simulated as if local object references. “liverefs” are intentionally left vulnerable to being severed when it comes to network abstractions which can break.

There are actually a bunch of interesting design decision tradeoffs here. Goblins takes the same approach as E (sturdyrefs/certificates rehydrated into liverefs, but then these can break and you manually reassemble when this occurs), but Waterken (studyrefs only) and Agoric (studyrefs/certificates, but store and forward as the default connection type) both also make some different choices. This would be worth detailing but I’m afraid it might grow rather large. A challenge will be to think about how to condense it.

It’s not written and I made the mistake of removing the TODO before publishing. @frandallfarmer and I had agreed it would be best to not leave TODOs in the document so I commented them all out, but given that this section has received so much commentary indicates we should have left some to just indicate which sections are incomplete. Oh well, promise pipelining section is being written, will be in a revision I will publish shortly this week.

Interesting to hear you say so. Originally the paper was going to be released without any examples and just highly compressed explanations of all the features provided, and then we decided to write two “short tutorials” on Goblins and then on Goblins as an ocap system, and these became the bulk of the document. It seems like we should really take the lesson that the code illuminates, have more of that, and integrate it.

The approach we are describing is NOT orthogonally persistent! It is representationally persistent, and includes the possibility of upgrade. Unfortunately while the code works it does not have clean syntax yet and so it may take longer to make this section have examples than everything else as we figure it out. Technically the core.scm of Goblins provides no support for serialization, but it is layered on top. We originally had this shipped as a separate project named Aurie, but then it turned out that Aurie was too commonly wanted, so we will be bundling it with Guile Goblins.

It’s clear this section isn’t being as clear as it should be though, and examples are badly needed.

That one for sure.

My favorite footnote in any CS paper ever is footnote 2 from The Art of the Propagator. It is only five words:

Equality is a tough subject.

The ultimate understatement!

Thank you for all the comments, this was great!

30

Also, I replied at length to @civodul; I haven’t had a chance to reply to everyone else yet, but am very happy with all the feedback so far. If I didn’t reply to you, know that I am making each and every piece of feedback an item to go through, so there’s a new revision coming out soon which should incorporate much of this, and I’ll include a changelog and will try to include whose comments prompted each change.

31

Hello! I haven’t gone through the doc with a fine-toothed comb, but I have a few things to say which are just broadly about the approach.

So I think part of the challenge with explaining this is that for readers like me, my instinct is to map the novel nature of Spritely/Ocapsz to my own programming experience, and then I struggle to find out what’s different about it, because in translating it, I feel like I’ve elided some subtle new feature of Spritely-style ocaps. So, for instance, I’ll take capabilities and turn them into additional incoming parameters of a function, and then go, okay, why don’t I just do that in general? What new ability am I gaining from a new function-dispatching syntax?

I think this speaks to the other challenge, which is separating the new semantics, the new syntax, and Guile’s own syntax (which will be new to many people). So the above is a challenge of separating the semantics of ocaps, with the new “$” operator, etc, and ways you might do that in another language. Then there’s stuff like define-values which is something which is novel for Guile (multiple values being uncommon outside of Lisp), but is not an aspect of Spritely.

2 Likes
32

Thanks for all the clarifications! It has me willing to learn more and to practice. :slight_smile:

Heh. It’s the fact that it has such an elegant and simple interface that makes it particularly inspiring to me. All the abstractions shown in the tutorial fall into place very nicely, it “feels right”.

1 Like
33

This one is true, and speaks to some of the comments @alanhkarp has made also. I’d like to elaborate more on the decisions made regarding language choice, noting also that it’s still true that we want multiple implementations. More soon.

1 Like
34

Yay! And I’m pretty sure you’ve been looking at Fibers for the sake of Shepherd, which as you know is a place I would actually like to see Goblins implemented, crossing over in Guix-land.

That’s a great and promising quote! It also helps answer some of the “why these language choices?” and I will say more soon.

1 Like
35

Sound like we need an architecture document that captures not only the decisions made but also the motivations for them. Sorry to recommend more writing for you @cwebber.

When I worked on the Itanium project, Intel had a very nice process. Each decision or change to a decision was captured in a document that had a fixed set of sections:

  • Summary of the decision
  • Requirement the decision addressed or the motivation for change
  • Stakeholders who might want to comment before the decision was finalized
  • Alternatives considered and reason for rejection

I found this last section to be the most useful.

It seems like a lot of work, but being able to understand the “why” of decisions dramatically reduced the thrashing that I’ve encountered on other projects.

2 Likes
36

When we have appropriate tooling, this will definitely be a thing. I’ve been a huge advocate of decision history since Habitat days…

I not sure if it will be a wiki or something else, but flat whitepaper isn’t the place to detail debates/options. Though those discussions should be linked-to/footnoted.

1 Like
37

I actually enjoy the writing quite a bit… and I like your suggested workflow. (MarkM did a great job of capturing the first, second, and fourth of your bullet points in his dissertation.) The main challenge is that the writing needs to be balanced with many other things that need my time right now. Life will feel manageable when it feels like I am working 2 or 3 jobs again, as opposed to right now where it feels like I’m working 20 or 30.

In the meanwhile, it would be good if some of this was captured kind of as a side effect. One approach to that is I’ve started recording some of the training calls on the Goblins internals I’m doing with @tsyesika, as of yesterday. Obviously that kind of video has a lot of overhead since it isn’t “curated”, but capture first, distill later (when possible), right?

38

I’ve done a new export of the paper, refresh! I’ve added a ChangeLog (though I need to retroactively detail some of the edits I’ve addressed from Alan’s feedback), but here’s the summary:

  • Several people asked about the promise pipelining part, which was previously incomplete, but now has been fleshed out!
  • I incorporated many of @alanhkarp’s comments, but I still have quite a few to go. Alan gave great and meticulous feedback, trying to get through all of it, hopefully will finish in next release
  • I got through all of @tsyesika’s feedback

More will happen soonish, and I’ll post here. Feedback on the promise pipelining section especially desired!

Thank you to everyone who’s commented so far! It’s been really helpful!

1 Like
39

I haven’t read the previous comments, so my apologies if some of this feedback is already covered.

Intro

I think everything in here is good, but the intro needs to be the elevator pitch to get someone to start/continue reading. Is there a way to punch it up? What will the reader get out of finishing the document? Will they be better programmers by learning about POLA? Will they learn that distributed programming isn’t inherently horrendously difficult? I would say so directly. What “takeaway gifts” are you giving the reader as a reward for reading? I usually like to imagine a very skeptical, argumentative, overworked, high-opportunity cost reader that I have to persuade, even for a technical paper. If they find something that doesn’t seem useful immediately, they’ll leave the page. Unfortunately, I think this model is probably accurate! People just don’t have the time, so we have to make them care. If we don’t do this, then the only people left reading are cultists who don’t really understand but want to be part of the club anyways.

Now we can see what bcom is: a capability specific to this object instance which allows it to change its behavior! (For this reason, bcom is pronounced “become”!)

I didn’t understand this part, so I think it could use more explanation.

interacitons with =fork-motors

^ just a typo

The promise pipelining section is very well-motivated and well-written. I think the average programmer reading it would understand the benefits of not having to wait for the roundtrip. You could maybe add a teaser about this in the intro!

This serialization mechanism is sealed off from normal usage; only the serializer can unseal it, preventing objects from interrogating each other for information or capabilities they should not have access to.16

I would explain further here that secrecy/privacy is not provided The memory can still be read at a lower level of abstraction. But importantly, access cannot be gotten (a reference cannot be forged) at this particular level of abstraction. I’ve found that this difference trips people up.

3.7. Distributed behavior and why we need it

I would move this section earlier (possibly reference in the intro as a teaser?). It’s very interesting and motivating for the rest of the paper. It shows how different the programming paradigm is from what people are used to. The technical details should stay this far down into the paper, though.

blogging example

Feel free to ignore this because I know you both have spent a lot of time on this example, but I think the blogging example does not have enough danger for it to be motivating to the average reader. It sounds like building a large amount of infrastructure, but the main danger would be that someone can approve a blog post or write something inappropriate. I think people might unfortunately conclude that ocaps are unnecessarily difficult - a kind of fussiness that move-fast-and-break-things types can ignore. But software can steal your money, steal your data, embarrass you, give you the wrong medication, wreck your business and everyone else’s if they depend on you, etc. The world runs on software, so the danger is extremely high! Previously, I think you (Christine) had the idea of using Mint as an example. I think people would find their data/money being stolen extremely motivating!

One thing that might be helpful in the blogging example is explaining it like a product (“Look! You know how you hate it when someone like Robert writes terrible stuff? Here’s how you can still let him write but rein him in.” Or “Lauren can do this, and this, and if this happens she can do this, and you can’t do that easily with ACLs!”), and then showing how exactly that is possible. Right now the “features” are intermixed with the particular code, which means the reader will probably under-appreciate them while trying to understand the code.

Overall, though, I think this is an incredible collection of knowledge and some of the best explanation I’ve seen of ocaps! I’m really excited to see this out in the world when you all are ready!

3 Likes
40

Thank you for all the comments @katelynsills! These are great. I will work on incorporating them in the next draft (where I hope to finish incorporating @alanhkarp’s as well).

You may be right. I was trying to think of a very social example. Money is a place where people feel things are high risk, and that’s thus a good motivating example… but I feel like most ocap examples are very financial oriented. This isn’t bad, but maybe there’s other risks that could be explored. I will have to think about it.