Cloud servers have a significant advantage over local setups: they're always on. This means that your Python scripts can run uninterrupted 24/7. In this post, we'll explore setting up a Python script to run periodically on 8 different cloud services.
Introduction
When scheduling a Python script to run periodically on a cloud server, there are three main components that we need to consider:
- The Python script itself
- Requirements (Python packages used in the script)
- Environment variables (API keys, etc.)
To demonstrate a complete example, we will use a script that uses the requests
package and one environment variable. This way, when we schedule the script on
various cloud services, we can see how to handle the requirements and environment
variables.
Python Script
import os
import requests
def main():
"""
Gets detailed information about a specified cryptocurrency.
The data includes the coin's ranking, supply details, and current price.
"""
coin_id = os.getenv("COIN_ID")
response = requests.get(f'https://api.coincap.io/v2/assets/{coin_id}')
coin_data = response.json()
print(coin_data)
return coin_data
if __name__ == "__main__":
main()
Requirements
requests
Environment Variables
COIN_ID=bitcoin
Cloud Services
Cron expressions are used to specify schedules on the cloud services presented below. You don't have to know anything about cron expressions to follow along. However, if you want to learn more about them, check out crontab.guru.
We will use * * * * * as the cron expression for most cloud services. This
expression will cause the script to run every minute. Some cloud services
don't allow running jobs as frequently as every minute. In those cases, we will
use a different cron expression.
GCP Cloud Functions
First, we are going to create a Cloud Function that will run our script. Then we will create a Cloud Scheduler job that will trigger the Cloud Function every minute.
Create a Cloud Function
Go to Cloud Functions and click "CREATE FUNCTION".
Give it a name, and select "HTTPS" as the trigger type.
Open the "Runtime, build, connections and security settings" dropdown and under
the "RUNTIME" tab locate "Runtime environment variables". Add COIN_ID variable
with value bitcoin. You can leave everything else as default and click "NEXT".
Set runtime to "Python 3.12" (or whichever Python version you want) and entry point to "main". Paste the following code in main.py:
import functions_framework
import os
import requests
@functions_framework.http
def main(request):
coin_id = os.getenv("COIN_ID")
response = requests.get(f'https://api.coincap.io/v2/assets/{coin_id}')
coin_data = response.json()
print(coin_data)
return coin_data
Add requests to requirements.txt so that it looks like this:
functions-framework==3.*
requests
We are done with the configuration. Click "DEPLOY".
It will take some time for deployment to complete. Once it is done, we can schedule our Cloud Function to run every minute via Cloud Scheduler. Copy the URL of the Cloud Function. We will need it in the next step.
Create a Cloud Scheduler Job
Go to Cloud Scheduler and click "CREATE JOB".
Give it a name and set frequency to "* * * * *" (every minute). Set timezone to whatever you want. Click "CONTINUE".
Set target type to "HTTP". Enter the URL of the Cloud Function that you copied in the previous step. For "Auth header" select "Add OIDC token". For "Service account" select "Default compute service account". Click "CREATE".
That's it. The Cloud Function will run every minute. To monitor runs, check the "Logs" tab of your Cloud Function.
AWS Lambda
First, we are going to create a Lambda that will run our script. Then we will create an EventBridge schedule that will trigger Lambda every minute.
Create a Lambda
Go to AWS Lambda and click "Create function".
Give it a name, set runtime to "Python 3.12" (or whichever Python version you want) and click "Create function".
Under the "Configuration" tab, locate "Environment variables" and add COIN_ID
variable with value bitcoin.
Even though Lambda has an inline code editor where we can write our script, it
doesn't allow us to indicate our requirements (in our case the requests
package). Therefore, we will have to create a zip file with our script and
installed Python packages and upload it to Lambda.
Create a new directory. Open your terminal and cd into it. Execute the following:
pip install requests --target .
This will install the requests package into the root of the directory. Then,
create a file called "lambda_function.py" and paste the following code into it:
import os
import requests
def lambda_handler(event, context):
coin_id = os.getenv("COIN_ID")
response = requests.get(f'https://api.coincap.io/v2/assets/{coin_id}')
coin_data = response.json()
print(coin_data)
Next, we need to create a zip file of the contents of the folder. If you are on Linux or Mac, you can do:
zip -r package.zip *
The name of the zip file is not important. However, the structure of the zip file is important. It should have a flat directory structure, with "lambda_function".py and installed package folders installed at the root:
package.zip
|- lambda_function.py
|- requests
|- urllib3
|- certifi
...
Go back to Lambda and upload the zip file.
Lambda is now ready to run our script. Next, we need to create an EventBridge schedule.
Create an EventBridge Schedule
Go to AWS EventBridge Scheduler and click "Create schedule".
Give it a name.
Under "Schedule pattern", select "Recurring schedule" for "Occurrence". Enter "* * * * ? *" (every minute) for "Cron expression". Select "Off" for "Flexible time window" and click "Next".
Under "Target detail", select "AWS Lambda" and under "Invoke" select your Lambda function. Click "Next".
Click "Next" on the next page and then "Create schedule" on the last page.
That's it. Lambda will run every minute. To monitor runs, check the "Monitor" tab of your Lambda and click "View CloudWatch logs".
AWS EC2
We need to start by launching an EC2 instance. Go to AWS EC2 and click "Launch instance".
Give it a name and select "Ubuntu" as the AMI.
Scroll down to create a new key pair. We will use this key pair to ssh into the instance. Give it a name and click "Create key pair". The key pair will be downloaded to your computer.
Click "Launch instance".
Now we need to ssh into the instance. For that, we need to know the public IP address of the instance. Go to the new instance's page and copy the public IP.
Open your terminal and cd into the directory where the the key pair was
downloaded. If you are on Linux or Mac, make sure that the key pair is not
publicly accessible by executing:
chmod 400 <key-pair-name>
Then, execute the following command to ssh into the instance:
ssh -i <key-pair-name> ubuntu@<public-ip>
Replace <key-pair-name> with the the name of your key pair and <public-ip> with public IP of your instance. For example:
ssh -i scheduler.pem [email protected]
Now we need to create script.py, .env for environment variables, and requirements.txt for requirements. We also need to create a virtual environment and install the requirements in it. Virtual environment allows us to install Python packages specific to our script without affecting the global Python installation.
Let's start by creating the virtual environment. First, we need to install the
python3-venv package:
sudo apt-get update
sudo apt-get install python3-venv -y
Then, we need to create the virtual environment and activate it:
python3 -m venv venv
source venv/bin/activate
Next, let's create requirements.txt so that we can install the requirements in the virtual environment. We will use Vim editor to create the file:
vi requirements.txt
This will open the editor. Press i to enter insert mode. Then, paste the
following:
requests
python-dotenv
Press esc to exit insert mode. Then, type :wq and press enter to save and
exit Vim.
python-dotenv is a package that allows us to read environment variables from
a .env file in our Python script.
Let's install the requirements in the virtual environment:
pip install -r requirements.txt
Now we need to create the .env file.
vi .env
Press i to enter insert mode. Then, paste the following:
COIN_ID=bitcoin
Press esc to exit insert mode. Then, type :wq and press enter to save and
exit Vim.
Finally, let's create the Python script:
vi script.py
Press i to enter insert mode and paste the following:
import os
import requests
from dotenv import load_dotenv
load_dotenv()
def main():
coin_id = os.getenv("COIN_ID")
response = requests.get(f'https://api.coincap.io/v2/assets/{coin_id}')
coin_data = response.json()
print(coin_data)
if __name__ == "__main__":
main()
Run the script manually to make sure that everything is set up correctly:
python script.py
It should print the data for Bitcoin.
We are going to schedule the script to run every minute. We will use crontab for that. crontab is a config file that describes the commands to run periodically. Let's open the crontab editor:
crontab -e
You will be asked to select an editor. Select 2 for Vim.
This will open the editor. Press i to enter insert mode. Use the arrow keys
to go to the last line and paste the following:
* * * * * . /home/ubuntu/venv/bin/activate && python3 /home/ubuntu/script.py >> /home/ubuntu/logs.txt
Press esc to exit insert mode. Then, type :wq and press enter to save and
exit Vim.
This will run script.py every minute and log the output to logs.txt.
use journalctl -u cron to check each run of the script.
After the first run, logs.txt will be created and you will be able to see the
output of the script. To view the logs, execute cat logs.txt.
We've successfully scheduled our script to run every minute. You can safely exit
the ssh session by executing exit. The script will continue to run in the
background.
DigitalOcean
We will create a Function on DigitalOcean by creating Function files locally and pushing them to DigitalOcean via doctl. We will create the following files: a Python script (__main__.py), a requirements file (requirements.txt), a build script (build.sh), and a DigitalOcean config file (project.yml). Only the config file will be in the root directory. The rest of the files will be in a packages/core/cryptocurrency_checker subdirectory.
The directory structure must look like this:
|- packages/
| |- core/
| | |- cryptocurrency_checker/
| | | |- __main__.py
| | | |- requirements.txt
| | | |- build.sh
|- project.yml
Start by creating the root directory.
Then, create project.yml and paste the following:
packages:
- name: core
functions:
- name: cryptocurrency_checker
runtime: 'python:default'
environment:
COIN_ID: bitcoin
Next, create packages directory. Within packages, create core directory. Within core, create cryptocurrency_checker directory. Inside the cryptocurrency_checker directory, create __main__.py and paste the following:
import json
import os
import requests
def main(args):
coin_id = os.getenv("COIN_ID")
response = requests.get(f'https://api.coincap.io/v2/assets/{coin_id}')
coin_data = response.json()
print(coin_data)
return {
'body': {
'response_type': 'in_channel',
'text': json.dumps(coin_data)
}
}
In the same directory, create requirements.txt and paste the following:
requests
Finally, create build.sh in the same directory and paste the following:
#!/bin/bash
set -e
virtualenv virtualenv
source virtualenv/bin/activate
pip install -r requirements.txt
deactivate
If you are on Linux or Mac, build.sh must be executable. To make it executable, run the following:
chmod +x build.sh
The files are ready. But, we need to install doctl and set it up. Install
doctl by following the instructions here:
https://docs.digitalocean.com/reference/doctl/how-to/install/.
Then go to Applications & API on DigitalOcean and generate a new token.
Make sure to check the "Write" box.
To authenticate, copy the token and execute doctl auth init. Paste the token
when prompted.
Next, we have to create a serverless namespace. Namespaces are used to group resources together.
doctl serverless namespaces create --label my_namespace --region nyc1
After creating the namespace, we need to connect to it:
doctl serverless connect
At this point, we are ready to deploy our files. Go back to the root directory (where project.yml is located) and execute the following:
doctl serverless deploy . --remote-build
Once the deployment is complete, we can schedule the function to run periodically.
To create a schedule, go to DigitalOcean Functions, open the namespace that we created, and click on your function.
Click on the "Triggers" tab and then "Create Trigger".
Give it a name, set the cron expression to "* * * * *" (every minute), and save.
That's it. The function will run every minute.
Heroku
Start by creating an new app on Heroku.
Under the Settings tab, locate and click on "Reveal Config Vars". Add
COIN_ID variable with value bitcoin.
We will be doing deployment via Heroku Git using the Heroku CLI. To install Heroku CLI, follow the instructions here: https://devcenter.heroku.com/articles/heroku-cli#install-the-heroku-cli
Once you have Heroku CLI installed, log in by executing heroku login.
We are ready to create and push files to Heroku Git. We will be pushing a Python script (script.py), a requirements file (requirements.txt) and a Heroku config file (Procfile).
The directory structure must look like this:
|- script.py
|- requirements.txt
|- Procfile
Start by creating the root directory.
Then, create script.py and paste the following:
import os
import requests
def main():
coin_id = os.getenv("COIN_ID")
response = requests.get(f'https://api.coincap.io/v2/assets/{coin_id}')
coin_data = response.json()
print(coin_data)
return coin_data
if __name__ == "__main__":
main()
Next, create requirements.txt and paste the following:
requests
Finally, create Procfile and paste the following:
script: python script.py
To push the files to Heroku, execute the following commands:
git init
heroku git:remote -a function-app
git add .
git commit -am "first commit"
git push heroku master
The app is now deployed. We need to schedule it to run periodically. To do that, we need to install the Cron To Go Scheduler add-on. Execute the following command:
heroku addons:create crontogo:free-trial
To set the schedule, go to your app on Heroku, and open the addon:
Click on "Add Job".
Set the schedule to "* * * * *" (every minute). Set the command to
python script.py and click "Add job".
That's it. The script will run every minute. To check the logs, execute the following in the terminal:
heroku logs --tail
Render
We will create a Cron Job on Render by connecting a GitHub repository. This repository will contain a Python script (script.py) and a requirements file (requirements.txt).
The directory structure must look like this:
|- script.py
|- requirements.txt
Start by creating the root directory.
Then, create script.py and paste the following:
import os
import requests
def main():
coin_id = os.getenv("COIN_ID")
response = requests.get(f'https://api.coincap.io/v2/assets/{coin_id}')
coin_data = response.json()
print(coin_data)
return coin_data
if __name__ == "__main__":
main()
Next, create requirements.txt and paste the following:
requests
We are ready to push the files to GitHub. Create a GitHub repository and follow the instructions on GitHub to push the files to it.
Now we need to create a Cron Job on Render by connecting to the GitHub repository that was just created. Go to Create a new Cron Job on Render. Select Build and deploy from a Git repository and click "Next".
Select the GitHub repository that you created and click "Connect".
Give your Cron Job a name, set the schedule to "* * * * *" (every minute),
and set the command to python script.py. Scroll down to set an environment
variable. Add COIN_ID variable with value bitcoin. Click "Create Cron Job".
That's it. The Cron Job will run every minute.
Railway
We will create a project on Railway by connecting a GitHub repository. This repository will contain a Python script (script.py), a requirements file (requirements.txt), and a Railway config file (Procfile).
The directory structure must look like this:
|- script.py
|- requirements.txt
|- Procfile
Start by creating the root directory.
Then, create script.py and paste the following:
import os
import requests
def main():
coin_id = os.getenv("COIN_ID")
response = requests.get(f'https://api.coincap.io/v2/assets/{coin_id}')
coin_data = response.json()
print(coin_data)
return coin_data
if __name__ == "__main__":
main()
Next, create requirements.txt and paste the following:
requests
Finally, create Procfile and paste the following:
script: python script.py
We are ready to push the files to GitHub. Create a GitHub repository and follow the instructions on GitHub to push the files to it.
Now we need to create a project on Railway by connecting to the GitHub repository that was just created. Go to railway.app and create a new project.
Select the GitHub repository that you created and click "Deploy now".
Now that the project is deployed, we need to add the COIN_ID environment
variable. Click on the Variables tab and add COIN_ID variable with value
bitcoin.
Next, we have to create a schedule. Click on the Settings tab and scroll down to "Deploy" section. Click on "+ Cron Schedule".
Railway allows for scheduling scripts to run every 15 minutes at most - not every minute. Therefore, we will use */15 * * * * (every 15 minutes) as the schedule.
That's it. The script will run every 15 minutes.
Vercel
We will create a project on Vercel by connecting a GitHub repository. This repository will contain a Python script (index.py), a requirements file (requirements.txt), and a Vercel config file (vercel.json). The Python script will be in an api directory. The requirements file and the Vercel config file will be in the root directory.
The directory structure must look like this:
|- api/
| |- index.py
|- requirements.txt
|- vercel.json
Start by creating the root directory.
Then, create vercel.json in the root directory and paste the following:
{
"redirects": [{ "source": "/", "destination": "/api" }],
"crons": [{ "path": "/api", "schedule": "0 0 * * *" }]
}
Notice that we are defining a cron job within the crons array. Vercel's Hobby plan allows for scheduling scripts once per day at most - not every minute. Therefore, we are using 0 0 * * * (every day at midnight) as the cron expression.
Next, create requirements.txt in the root directory and paste the following:
requests
Finally, create api directory inside the root directory and create index.py inside the api directory. Paste the following into index.py:
from http.server import BaseHTTPRequestHandler
import os
import requests
class handler(BaseHTTPRequestHandler):
def do_GET(self):
self.send_response(200)
self.send_header('Content-type','text/plain')
self.end_headers()
coin_id = os.getenv("COIN_ID")
response = requests.get(f'https://api.coincap.io/v2/assets/{coin_id}')
coin_data = response.json()
print(coin_data)
return coin_data
We are ready to push the files to GitHub. Create a GitHub repository and follow the instructions on GitHub to push the files to it. Make sure to initialize the repository in the root directory.
Now we need to create a project on Vercel by connecting to the GitHub repository that was just created. Go to vercel.com and create a new project by importing the repository.
When the repository is imported, you will be able to configure the project. The
only thing that we need to do is add the COIN_ID environment variable with
value bitcoin and deploy.
That's it. The script will run every day at midnight.
