Hey, I'm Marco and welcome to my newsletter!
As a software engineer, I created this newsletter to share my first-hand knowledge of the development world. Each topic we will explore will provide valuable insights, with the goal of inspiring and helping all of you on your journey.
In this episode, I want to show you how I created a system that lets you share posts from the same website multiple times on HackerNews.
You can download all the code shown directly from my Github repository: https://github.com/marcomoauro/hackernews-automation
DISCLAIMER
While it may seem intriguing, it's important to recognize that trying to bypass soft bans or manipulate platforms like HackerNews is unethical and violates their terms of service. This post is purely for educational purposes, undertaken as a personal challenge following Implementing's soft ban. However, I strongly advise against such actions due to the potential legal and reputational consequences.
👋 Introduction
A few weeks back, I found a really interesting post by
on Substack. He explained how he promotes his posts, including sharing them on platforms like HackerNews. Curious, I looked up HackerNews and discovered it's a community focused on IT and entrepreneurship, run by Y Combinator.In Gregor's post, he talks about how one of his posts made it to the top page, bringing him tons of traffic and new subscribers. That got me excited, so I decided to start sharing my posts there too.
On my second attempt, I published the post around 8 am. While I was riding my scooter 🛵, my phone kept vibrating non-stop. I pulled over and found numerous emails from new subscribers. My post made it to the top page!
Looking at the newsletter statistics, my post totals 5k+ visits and 20 new subscriptions in about one day!
Strategy
After achieving quick results, I decided to post every day. However, after about 10 posts, none of them were receiving likes or visits, making me question whether the topics were engaging enough. After 10 days of this, I started to doubt myself. So, I decided to investigate further using a private browser. To my surprise, I discovered that my new posts weren't appearing at all on the page for new posts.
I checked and read in other posts that while HackerNews allows publishing multiple posts from the same site, it doesn't show these posts in the dedicated feed, meaning nobody sees them.
I couldn't use the Implementing URL anymore.
That's when I switched to my engineering mindset, embraced the challenge, and started searching for a solution.
💡 Solution
Let's summarize the solution I've come up with using this sequence diagram.
I thought of adding an intermediate Frontend. This way, I could use its URL for HackerNews posts and switch it up using Netlify's API, the hosting service for my frontend. That would solve the issue of having the same URL in all posts.
The goal of this frontend is to contact my backend via API by sending it the path after the first "/", then the backend searches the database where I set up a table with the format <path> => <original post on Implementing>. Once it finds the Implementing URL, it simply redirects to the newsletter.
After using it for two weeks to publish five posts a day, I can confidently say it works!
Systems
I implemented these 3 components:
Frontend: calls the backend and manages redirection to Implementing.
Backend daemon: takes care of publishing new posts on HackerNews with the help of ChatGPT.
Backend Api: retrieval of the original post from a specific path passed by the frontend.
🌐 Frontend
I didn't know much about it, so I asked my girlfriend, who specializes in the Frontend stack. We set up a Vue.js template that's easy to tweak and can be quickly deployed on Netlify.
We set up a router file that directs all routes to the same component. When the component is loaded on the webpage, it extracts the specified route after the first "/", sends a request to the backend API with the path parameter in the query string, receives the final URL, and then redirects accordingly.
Are you interested in the frontend Vue.js template and how I deploy it to Netlify? I will be publishing a post about it soon. Sign up so you don't miss it!
⚙️ Backend
As always for the backend, I begin with my Node.js backend template, which I covered in this episode:
Database
I set up a MySQL database using the free tier of Heroku's JawsDB add-on. Since I don't expect a significant increase in records, I'm comfortably staying within the limits of this plan.
I created 3 tables that represented:
posts: this entity references the original post on Implementing:
path: configurable route stored in the database to be inserted after the "/" in the frontend URL.
url: URL linking to the post on Implementing for the final redirection.
is_pickable: a flag used to exclude the post from being published on HackerNews.
submissions: this entity represents the publication of an Implementing post on HackerNews. The same post can be published multiple times (1:N relationship):
title: the title of the HackerNews post (generated by ChatGPT).
url: the final URL, combining the frontend domain and the configured post route.
text: a description of the HackerNews post (generated by ChatGPT).
site_titles: this entity represents a new subdomain of the frontend:
title: string used as the subdomain.
Daemon
Now, let's make a daemon that regularly sends new posts to HackerNews. We can show how it works with this sequence diagram:
For randomly selecting a post and generating the title and text, I made this function computePost:
const computePost = async () => {
const [post] = await query(`
select *
from posts p
where is_pickable = true
and exists(select * from submissions s where s.post_id = p.id)
order by rand()
limit 1
`)
const [{title: old_title, text: old_text}] = await query(`
select title, text
from submissions
where post_id = ?
order by rand()
limit 1
`, [post.id])
const old_titles = await query(`
select title
from submissions
where post_id = ?
`, [post.id])
let post_title = await completionByAI({
system_message: `rephrase this title in English to create a post on HackerNews. It must be creative, click-inducing and different from the following titles: "${old_titles.join('; ')}"`,
user_message: old_title,
system_message2: 'answer in English with only the title'
})
post_title = post_title.replace(/"/g, '')
const post_text = await completionByAI({
system_message: 'rephrase this comment in English to create a post on HackerNews:',
user_message: old_text,
system_message2: 'reply in English with just the comment'
})
return {
post_id: post.id,
post_title,
post_path: post.path,
post_text
}
}
It fetches a random post by sorting them using the MySQL rand() function. Then, it generates the title and text using the completionByAI method, which utilizes the ChatGPT completion API by passing it to the prompt system and user inputs.
Next, the daemon calls the function changeFrontendSiteName, which uses the Netlify API to change the sub-domain of the frontend application. It generates a new random sub-domain using the nanoid library.
const changeFrontendSiteName = async () => {
const body = {
name: nanoid(10)
};
const {data: {default_domain}} = await axios.patch(`https://api.netlify.com/api/v1/sites/${process.env.NETLIFY_SITE_ID}`, body, {
headers: {
'Authorization': `Bearer ${process.env.NETLIFY_OAUTH_TOKEN}`,
'Content-Type': 'application/json'
}
})
return `https://${default_domain}/`
}
The site ID and authentication token are fetched from the environment variables.
We now use the Puppeteer library to set up a headless browser. We'll emulate login operations and post submission this way, as HackerNews allows it via the public API.
const loginHackerNewsAndPublishPost = async ({post_title, post_url, post_text}) => {
const browser = await puppeteer.launch({ args: ['--no-sandbox'], headless: true});
const login_page = await browser.newPage();
await login_page.goto('https://news.ycombinator.com/login');
await login_page.waitForSelector('input[name="acct"]');
await login_page.type('input[name="acct"]', process.env.HACKERNEWS_USERNAME);
await login_page.waitForSelector('input[name="pw"]');
await login_page.type('input[name="pw"]', process.env.HACKERNEWS_PASSWORD);
await login_page.waitForSelector('input[type="submit"]');
await login_page.click('input[type="submit"]');
const submit_page = await browser.newPage();
await submit_page.goto('https://news.ycombinator.com/submit');
await submit_page.waitForSelector('input[name="title"]');
await submit_page.type('input[name="title"]', post_title);
await submit_page.waitForSelector('input[name="url"]');
await submit_page.type('input[name="url"]', post_url);
await submit_page.waitForSelector('textarea[name="text"]');
await submit_page.type('textarea[name="text"]', post_text);
await submit_page.waitForSelector('input[type="submit"]');
await submit_page.click('input[type="submit"]');
}
here's a little demo of how it works, it's always nice to see it in action :)
Finally, I persist new records in the database indicating a new submission of the post and a new record for changing the frontend domain.
API
We want to create an API that, given a string representing the route of the frontend redirect URL, finds the original post in the database and sends back its URL to the client.
We define the new src/controller/post.js controller:
import log from "../log.js";
import Post from "../models/Post.js";
export const getPostUrlByPath = async ({path}) => {
log.info('Controller::Post::getPostUrlByPath', {path});
path &&= path.split('?').at(0);
const post = await Post.getByPath(path);
return {url: post.url}
}
The "path" field may include a query string. To preserve this information, I perform the trimming operation on the backend instead of the frontend. This way, we avoid losing this data during logging. It could prove useful for statistical purposes or to track the source of visits. It's always better to have more information than less!
we create the src/models/Post.js model that will interface with the database:
import log from '../log.js'
import {query} from '../database.js'
import {APIError404} from "../errors.js";
import memoize from "../memoize.js"
export default class Post {
id
path
url
constructor(properties) {
Object.keys(this)
.filter((k) => typeof this[k] !== 'function')
.map((k) => (this[k] = properties[k]))
}
static fromDBRow = (row) => {
const post = new Post({
id: row.id,
path: row.path,
url: row.url,
})
return post
}
static getByPath = memoize(async (path) => {
log.info('Model::Post::getByPath', {path})
const rows = await query(`
select *
from posts
where path = ?
`, [path]);
if (rows.length !== 1) throw new APIError404('Post not found.')
const post = Post.fromDBRow(rows[0])
return post
},
{
promise: true
})
}
The getByPath method looks up the post by querying the database with the path. If it finds a match, it converts the record into a DTO. I've also applied memoization to cache the result in memory. I don't foresee scalability issues since the cache can store a maximum number of instances equal to the number of posts in the database, which is around 10 elements. If scalability becomes a concern, using a system like Redis might be a better option.
Finally, we create the new api in the src/router.js file:
import {getPostUrlByPath} from "./controllers/post.js";
router.get('/posts/get-url-by-path', routeToFunction(getPostUrlByPath));
You can download all the code shown directly from my Github repository: https://github.com/marcomoauro/hackernews-automation
📊Results
Since activating the daemon, I've observed a significant increase in visits compared to before. It's possible that this trend isn't solely due to the daemon, as the visits from the post that made it to the top page certainly contributed as well.
I also keep track of the visits I receive through the system by collecting backend logs. For every visit on the redirected frontend, I receive an API call to the backend, which is logged. On average, I receive between 100 and 200 invocations per day.
And that’s it for today! If you are finding this newsletter valuable, consider doing any of these:
🍻 Read with your friends — Implementing lives thanks to word of mouth. Share the article with someone who would like it.
📣 Provide your feedback — We welcome your thoughts! Please share your opinions or suggestions for improving the newsletter, your input helps us adapt the content to your tastes.
💬 Chat with me — If you have any doubts or curiosity, please write to me, I will be happy to answer you!
I wish you a great day! ☀️
Marco
Haha, very interesting read! I like the way you took the problem to your heart and solved it! I'd say if you haven't reached out to hacker news, try doing so. They are great folks and might actually pin your newsletter there for a week when you let them know of this trick!
That's interesting! Thanks for sharing. Do you have to create a new title each time you post on Hacker News, or else your message won't reach a larger audience?