README.md

Lithops configuration

By default, Lithops uses the localhost backend when no configuration is provided. To run workloads in the cloud, you must configure both a compute and a storage backend. If either backend is misconfigured, Lithops will not be able to submit workloads. Configuration can be provided in a configuration file or at runtime via a Python dictionary.

Configuration file

To configure Lithops through a configuration file, you have multiple options:

1. Create a new file called config in the ~/.lithops folder (i.e., ~/.lithops/config).

2. Create a new file called .lithops_config in the root directory of your project from where you will execute your Lithops scripts.

3. Create a new file called config in the /etc/lithops/ folder (i.e., /etc/lithops/config). Useful for sharing the config file on multi-user machines.

4. Create the config file in any other location and set the LITHOPS_CONFIG_FILE environment variable:

   LITHOPS_CONFIG_FILE=<CONFIG_FILE_LOCATION>

Configuration at runtime

An alternative way to configure Lithops is to use a Python dictionary. This lets you pass configuration details as part of the Lithops invocation at runtime. The full list of sections and keys is here.

Compute and Storage backends

Choose your compute and storage backends from the table below.

Compute Backends

Storage Backends

Localhost Serverless (FaaS) Backends: AWS Lambda Google Cloud Run functions Azure Functions Aliyun Functions Oracle Cloud Functions OpenWhisk Serverless (CaaS) Backends: IBM Code Engine AWS Batch Google Cloud Run Azure Container Apps Kubernetes Knative Singularity Standalone Backends: Virtual Machine IBM Virtual Private Cloud AWS Elastic Compute Cloud (EC2) Azure Virtual Machines Google Compute Engine

Localhost Object Storage: IBM Cloud Object Storage AWS S3 Google Cloud Storage Azure Blob Storage Aliyun Object Storage Service Oracle Cloud Object Storage Ceph MinIO OpenStack Swift In-Memory Storage: Redis Infinispan

Verify

Test if Lithops is working properly:

Using Lithops configuration file

import lithops

def hello_world(name):
    return f'Hello {name}!'

if __name__ == '__main__':
    fexec = lithops.FunctionExecutor()
    fexec.call_async(hello_world, 'World')
    print(fexec.get_result())

Providing configuration at runtime

Example of providing configuration keys for IBM Code Engine and IBM Cloud Object Storage:

import lithops

config = {
    'lithops': {
        'backend': 'code_engine',
        'storage': 'ibm_cos'
    },
    'ibm': {
        'region': 'REGION',
        'iam_api_key': 'IAM_API_KEY',
        'resource_group_id': 'RESOURCE_GROUP_ID'
    },
    'ibm_cos': {
        'storage_bucket': 'STORAGE_BUCKET'
    }
}

def hello_world(number):
    return f'Hello {number}!'

if __name__ == '__main__':
    fexec = lithops.FunctionExecutor(config=config)
    fexec.map(hello_world, [1, 2, 3, 4])
    print(fexec.get_result())

Summary of configuration keys for Lithops

Group	Key	Default	Mandatory	Additional info
lithops	backend	aws_lambda	no	Compute backend implementation. localhost is the default if no config or config file is provided.
lithops	storage	aws_s3	no	Storage backend implementation. localhost is the default if no config or config file is provided.
lithops	data_cleaner	True	no	If True, automatically deletes temporary data written to storage_bucket/lithops.jobs.
lithops	monitoring	storage	no	Monitoring backend. Built-in: storage, rabbitmq, redis, aws_sqs, gcp_pubsub, azure_queue.
lithops	monitoring_interval	2	no	Interval in seconds for monitoring checks when using storage monitoring.
lithops	data_limit	4	no	Maximum size (in MB) for iterator data chunks. Set to False for unlimited size.
lithops	execution_timeout	1800	no	Maximum execution time in seconds for functions. Functions exceeding this time are terminated. Can also be set per call via the timeout parameter.
lithops	include_modules	[]	no	List of dependencies to explicitly include for pickling. If empty, all required dependencies are included. If set to None, no dependencies are included.
lithops	exclude_modules	[]	no	List of dependencies to exclude from pickling. Ignored if include_modules is set.
lithops	log_level	INFO	no	Logging level. Options: WARNING, INFO, DEBUG, ERROR, CRITICAL. Set to None to disable logging.
lithops	log_format	"%(asctime)s [%(levelname)s] %(name)s -- %(message)s"	no	Format string for log messages.
lithops	log_stream	ext://sys.stderr	no	Logging output stream, e.g., ext://sys.stderr or ext://sys.stdout.
lithops	log_filename	(empty)	no	File path for logging output. Overrides log_stream if set.
lithops	retries	0	no	Number of retries for failed function invocations when using the RetryingFunctionExecutor. Default is 0. Can be overridden per API call.