2024 Jopro Development Journal

Step chunks are working! Though I'm just noticing a bug in new step chunks.

The step chunk content isn't an actual template yet, I just wanted to see if I could get multiple steps. One thing that was tricky was that swapping out the TipTapEditor seemed to not change the content, so I had to render all the editors at once then swap them. Overall looking good, I need to tweak some things for sure, and maybe if there's more than 4 steps I should display the number of steps. So for example if I end up doing that survey for happiness like a PERMA survey, that has say 40 questions I could display the number or progress. Partial progress is always saved. I need to ensure clicking next also saves the progress since it will only actually save progress when you're typing. But overall it's a big feature to get out since it's important to a lot of templates I want to add!

🐞 To be fixed, the date has disappeared from the chunks. I did a bunch of refactoring of the chunks, so I need to make sure they get back in.

A few thoughts crossed my mind this morning while doing my weekly win update. I think moving the syncing status indicator to the toolbar would be nice. I think the chunks would feel cleaner. The other thing is that it would be really cool if I could maintain a weekly wins journal but add things in as they happen each day. At the end of the week (if I don't entry from within 24h or something like that), then I'll send out a summary of weekly wins generated with AI from the weeks entries. I need to think about how to generalize all these ideas I have for analytics, so that when I implement it it makes sense. Though I want to avoid over optimizing right now just to get that feature out. But I think it would be the hallmark feature of a Weekly Wins protocol.

Another thing is that I transform all uploaded images to WebP, in particular including the profile photos. Seems like transparent PNG images are losing their transparency. I also think the style of the web links could be better.

Also as I mentioned earlier I really need to update image security by sending all requests through the server rather than directly serving from S3. I think I'd like to redo the paths to include the user/notebook id in the object path, but I'm not entirely sure how to do with transloadit. Though I know I can override steps (or just all the steps) in the config that I send with the requests.

Chunk refactoring remains elusive but I'm interested in doing it today and I don't have anything else going on in particular.

Something else that's bothering me is that on published sites the date input is coming up as usable by last pass as a credit card? For example on a8b

Running into some annoyances with the step chunk stuff. The way the thing has been set up is a little problematic. I need to move saving logic up out of the editor into the chunk level. Shouldn't be much of an issue since the setContent call already happens. We can just hook into this to call into the server and set the entire state. I should be able to have stepped chunks working soon enough, I think I'm just preoccupied with some personal things at the moment. Stepped chunks will really enable EMDR which will be awesome as a feature to have. I think I'm hesitant because I actually want to refactor some of the chunk code, which I know is a slightly bigger deal that I don't want to start this late (it's 2am).

Finally getting the tiles to generate for opengraph! It took a bit of wrangling.

arn:aws:lambda:us-east-1:098144166132:layer:node-chromium:3 is the layer ARN for anyone interested. It uses "@sparticuz/chromium": "^121.0.0" and "puppeteer-core": "^21.9.0".

import puppeteer from "puppeteer-core";
import chromium from "@sparticuz/chromium";

export const handler = async (event, context) => {
    chromium.setHeadlessMode = true;
    chromium.setGraphicsMode = false;

    const browser = await puppeteer.launch({
        args: [...chromium.args, "--hide-scrollbars", "--disable-web-security"],
        defaultViewport: chromium.defaultViewport,
        executablePath: await chromium.executablePath(),
        headless: chromium.headless,
        ignoreHTTPSErrors: true,
    });

// ...
}

The code has something a little like above. As a note to myself it would be quite nice to get code highlights in that code block. With opengraph working I also implemented the rewrite strategy that looks at hosts. I'll go ahead and see how that works if at all, just waiting for DNS updates. I'm still trying to redirect it through cloudfront with the cloudfront served certificate but I could also bind a8b.io directly to the Vercel project.

The blog is served from the root domain! Amazing. I ended up doing a combination of the rewrites in next config and a custom middleware. I might actually move it all to middleware and I can quite easily actually scale this to other domains if I want by making very simple cloudfront fronts for them. The main issue I ran into is that rewrites from "/" don't seem to take effect? I had to write a custom middleware that looked something like this

import { NextResponse } from 'next/server'
import type { NextRequest } from 'next/server'
 
export function middleware(request: NextRequest) {
    const notebookAlias = request.headers.get('X-Notebook');
    if (request.nextUrl.pathname === '/' && notebookAlias === "a8b") {
        return NextResponse.rewrite(new URL('/journals/web/1/2d6843d0-137d-439e-9442-1f1a048cebd8', request.url))
    }
}

export const config = {
  matcher: '/',
}

The nice thing is that I can scale this based on resolving the alias very easily. So anyway that's live at https://www.a8b.io now. The only thing I want to fix now is that the opengraph image is a bit blurry. I think I'll make it a PNG image instead if possible and maybe upscale the whole thing including the HTML to 1200 width as expected as minwidth for opengraph images.

The image does look pixelated and the border is cut off making me think the aspect ratio is actually slightly off. In any case that's for another time. For now I'll leave it as is, as I think building EWI and other use cases is the most important thing.

A nice thing to do would be allow the super cool scrollbar I have to work on the public published notebooks.

Notification didn't send via email for recent journal notification that triggered for my EWI journal about an hour ago. The notification is showing up on the dashboard.

I decided to try to build a standalone lambda to generate screenshots of the preview HTML, but even then I am having issues making the layer work. I tried using this lambda layer ARN to get things working but it seems incomplete as I can't use chromium without puppeteer core as a module and I can't import that so I'm not sure what's going on. The struggle bus continues on its route. I also tried doing the whole thing myself but the package is bigger than 50MB which is the max for lambda. The other option is I make the layers myself, which I will have to do tomorrow. Generating the snapshot response object seems easy enough from a lambda with a function URL. I can return the base 64 encoded response that lambda itself will unpack as the response body in the HTTP response.

Unrelated but custom domains work. There wasn't much to it other than a few more edits to the URL construction logic. The main issue now is that I'm getting a 308 redirect for some reason when I hit the home page.

I'm not sure who it's coming from though I don't see any 308s from Vercel. But in any case it redirects from the root of the site into the full path that I append on the origin request. Interestingly this doesn't happen with live.jopro.app. In any case www.a8b.io now points to the live journal. I would love to migrate some of the old content. Alas for now I'm stuck in solving small issues. I fear there will be more complications if I try to figure out how to fix a8b.io itself to the new domain, rather than just the subdomain. If I have to create an A record I'll need a fixed IP and probably some custom service to do the resolution. Sounds messy.

I would 100% pay to have an API that handles all this for me, an endpoint with some sort of SNI proxy that I can CNAME and also point an A record to, that I can configure programmatically via some API calls. That would also handle certificate issuing and validation and have an API to also surface the DNS settings that you can display in the user profile to verify. The only problem is, though I would pay for it I'm not sure it's that common a thing or would have that many customers. Still, if companies like substack are charging $50 to set it up, it must be valuable. I could easily see it being worth $100 a month or more for the developer. I say that because there's no easy ways to do this. Getting a static IP I think costs money (costs a lot for cloudfront anyway) and to run something like an SNI proxy I'd need some dedicated instances running with good network bandwidth, which also costs a lot of money every month. So the prospect of expensive and easy is great compared to expensive and hard as the alternative. If I get it somewhat working for jopro in the future I may look to generalize it.

In any case I seem to be creating a lot of problem for myself by going down these more complicated paths when I could have been working on the stepped chunk editor that I need to do. Still there's something satisfying when I can properly get the root domain to work, and when the opengraph cards work. The hosted notebook will look super awesome and I will be glad to share content from it. That being said I was hoping I would be further today. I think the opengraph images will be conquered tomorrow since I know the solution. And I'll let the 308 problem sit on the backburner and move on to the step editor and back to the gratitude protocol.

In other news there's a potential bug in the search result formatting on mobile when the thing doesn't wrap (need to hide overflow in think) and also asking questions seems to error out on the client side e.g. the open ai integration potentially has a problem.

I have a solution to fix the 308 issue. I think I can use rewrites with a host header check to wire the root path to the appropriate journal. I may add a entries or posts/ part to the URL as well to make routing for the entries easier.

This isn't super scalable, but it would work for some one offs and I'm not so concerned about the scalability of that at the moment without users wanting to use it. I could always invest more if it's an important feature.

A scalable option for the future would be to redirect all non jopro.app host requests to a virtual-host/ subpath that then inspects the host header on the request and serves up the correct notebook.

I ended up adding some more web publishing features today despite me saying previously that I wasn't going to. I did so because I decided to start hosting a8b.io on Jopro! Which will be really fun

There's meta descriptions for the journal itself and profile images, author names, bio taglines, links. The links are really cool, you just paste the link url and can optionally specify the text. If you don't included the text then it will use an icon! The icons come from https://www.npmjs.com/package/react-social-icons so they automatically adapt to the network and I can just style them to match the page. I think it looks really clean and I don't have to have specific input boxes for specific link types. I use a similar editor the tag editor.

I'm trying to support custom domains but I think it might be a little complex. I think at the moment I need to create a new cloudfront distribution for each domain unless I am able to do some remapping from some front end in front of the cloudfront distribution and attach the certificate there and do SNI. That might be a lot more involved than I have time for at the moment to create a solution or host out something like this https://github.com/ameshkov/sniproxy

This idea is kind of out of left field, and isn't directly related to Jopro but a problem I have is that I need to generate open graph / meta images so that when I share posts like this one https://live.jopro.app/a8b/2024-02-19T02:09:16.946+00:00 to LinkedIn for example that I have a nice little card that shows the post title for example or the author profile picture or a preview of the website. Ideally it will look cool and be catchy. Kind of like the post cards I made manually in Canva when I was doing skincareaddict. If I develop this for Jopro there might be others who would want to use something like that as an API for their websites or as a plugin for Wordpress. I'll spend some time tomorrow I think looking in to what exists out there!

I spent some time working up a design for the opengraph image that I think would look cool.

Which is as above. I think it looks really slick! I just need to figure out how to take this rendered HTML and turn it into an image on the server in the metadata chunk.

I tried using ImageResponse but it's severely limited. I can't even get images to render inside of it, let alone my fancy gradient with absolute positioning. Everything has to have display: flex on it and you can't render numbers without converting them to strings 😒 overall it's really disappointing especially since I spent like an hour wrangling the JSX component to try to make it conform.

Locally I can use capture-website which runs on puppeteer and chromium, but it has issues loading in vercel. I'm not sure I want to spend a lot of time debugging it, but on the other hand I imagine I will need to make it work for tests anyway. I found chrome-aws-lambda which has some promise but I'm having issues getting it to work in Vercel. I may end up writing my own Lambda function to do this. I'm excited to get this working but how annoying that there isn't a simple way to do this.

I think I'm going to leave it for now since it's causing a lot of headache.

I found another issue on mobile. That the copy / paste mobile menu tends to put itself over the italic/bold tooltip/bubble menu. We may have to override the behaviour on mobile.

Also feature request, a toggle for enable disable spell check. This is a feature in the demo editor.

🧠 Weekly summaries as emails would be really cool for the dev journal.

Now that I'm using Jopro for a8b.io i'd really like to be able to have user data that I can add to the top of profile like an author profile link and links to other sites/social media/etc.

Reminders are now working! I made them super simple. They alert you at X intervals (every day/week/month/etc), whatever frequency you set in the notebook settings. Notifications now show up on the dashboard page as well as in the user bubble as pictured.

The notifications also send as emails. The emails and notifications trigger once when more than the interval amount of time has passed since the last notebook entry and the last notification. So for example a daily notebook would notify you the following day at the same time (to the nearest hour) and then would alert you again every one day since the last alert. This allows you to skip an interval and still get alerted, but the alerts aren't on a fixed schedule.

Overall feeling good about the scheduler. This approach allows me to gloss over certain details like time zones. The approach isn't 100% ideal though for all cases, but I think it actually is quite useful most of the time. And I can continue tweaking it, for example I can take off an hour from the time to make it so that the reminder doesn't come after the time you would have already done the thing. e.g. if you tell yourself you'll do it at 8pm monday every week you don't want the reminder at 9pm, you don't want to keep pushing it later and later with subsequent reminders that come after the allotted time.

Clicking the little X next to the notification dismisses it. I was going to have some more complex system but I think this is probably the easiest way and fairly robust and straightforward.

All emails are managed by resend and react email, and the scheduler lambda actually calls into the next.js server (although the API is actually serverless functions) to do actual email sending.

From using on mobile, first there's an issue with the create notebook form again. I think this is the first thing that really needs tests. The formatting feels squashed on mobile and to be fair I've never tried to create a notebook on mobile. The template selector takes a comical amount of space because the EWI description is too long. Also the field labels and inputs need space between

Image uploads need to insert a trailing space on mobile at least. The error message thing could just be moved to next to the save button. Don't necessarily see why it needs to be on top with an auto scroll unless you start putting the errors next to the fields (which eventually I'll do).

I've been thinking a lot about how to pitch Jopro and I think I'm getting some clarity. Will invest some more time into the positioning. Overall having a really good time building this product ☺️

Truncate journal titles to ensure they fit the screen or wrap them in the User bubble. Review image security! Right now everything gets publicly hosted. Access should be through the API secured by session unless the journal is public. The API can also handle that logic.

I think I want to start pushing towards a product launch, for quite the silly reason. I want to make an april fool's joke. I have an idea for it. But it's kind of a waste if there isn't a real product behind it. So I'm hoping to launch by mid march to give me some time to make the joke site / video. Here's what I think I'd be happy with putting out to the public.

EWI editor that works really well
Free form editor that works well
Gratitude protocol
Dream journal

I think I'm actually pretty close, at least in terms of the infrastructure. I think most of the work will be well researched articles to understand the best way to implement the gratitude and dream journal. There will be a lot of bits and bobs to actually launch like hooking up payments and subscriptions and figuring out how to do features per tier of subscription e.g. web hosting on plus only.

Email notifications are working! I'm using resend with react emails and it's great. I can use tailwind right in the email. I'm not sure about best practices with using flex layouts in an email, so I've been sticking to using the Column/Row components that come with react email.

The emails are looking great so far! I'm working on scheduling reminders at the moment, which will enable users to reminder themselves to add entries to adhere to a protocol. I think for now I'm going to have a simple scheduling option, with one parameter which is the frequency. I was going to have complicated scheduling but I think that's a pain to set up from a user perspective and most people probably don't want that, they just want to get themselves to do the thing. The way I'm thinking about doing it is as follows

You set a frequency, if the last entry in the notebook is more than the frequency then you get a notification to do the notebook.
At the start of the trigger day you get a reminder in the morning (local time, I need a way for users to set their time zones as well as email preferences for notifications). If you complete an entry before the end of the day that's all the reminder you get.
If you don't add an entry, you get another reminder at the end of the day asking you to complete an entry for the notebook.
I'm not sure exactly how many follow ups to send. For now I'm thinking just those two entries as the default option. But I can see how in reality sending additional emails each day until the user adds an entry could be desirable for some users. I may add a 2nd parameter to enable that functionality.

I fixed the bug I mentioned earlier where the title and description doesn't load for the live journals. This was a bug in the way the URL parameter was decoded.

I made a small optimization to use JWTs for next auth sessions. I think this will help a lot to reduce SQL compute time which has been high for no real reason (reached 75% of limit and I'm the only person using it) and also this may help with a weird issue I get sometimes where the SQL pool will timeout and this will sometimes make it look like the user is logged out even when the session is valid. For example the UserBubble in the bottom right will say sign in, but when you refresh it will show you as signed in. So this is a transient issue, and when I look at the logs it shows a SQL timeout. 🤞 Hopefully this solves the issue.

Had a bit of a "lazy" day on Wednesday. I started on email development, but overall didn't do much coding other than small fixes. I did think a lot about Jopro though, and had some interesting ideas about storing survey results for some things like Authentic Happiness or PERMA scores in notebooks which I noted in the previous entry. I think this would be a cool feature and very well aligned. I refactored the "Therapeutic Writing" information hub into Writing Protocols. I think this is more aligned to the loose definition of what's going on, for example with the gratitude protocol, even though they will still be writing based.

I also had a small idea for trying to change the toolbar, moving the journal title onto the page and then moving the setting alongside open/search like it is on mobile and then having a tiny floating menu bar that only is on the left! I liked the idea but I think from a functionality perspective it's important to have the notebook title on the menu bar and available as you scroll.

Another small thing that crossed my mind just now is that the open notebook modal can be taller in height than the window and it results in something like this.

I would like to cap the height of the modal around the view height and then put scrolling on the inner notebook select frame, but persisting "Create New Notebook".

Something that occurred to me on the train was that I originally said I was going to allow for a reset time of around 4AM rather than midnight. This will allow for things I write early in the morning still to get associated to the "day" that was happening. The midnight thing I think is more of a technicality, and I'd rather the writing I do for Wednesday be associated with Wednesday even if it's Thursday at 1AM for example.

Today during the working day (I always end up writing these early in the morning after the working day is done) I will work on those 3 things I outlined I think, email, chunks with steps/state, and editable prompt data. At some point I also need the ability to enable renaming the chunks, and creating new chunks on demand. The chunks will have names appropriate for the notebook template. For example in EMDR a chunk may be called a session.