Skip to content

READ_S3

The READ_S3 node takes a S3_key name, S3 bucket name, and file name as input, and extracts the file from the specified bucket using the S3_key that was saved.Inputs ------ default: NoneParams:s3_name : strname of the key that the user used to save the access and secret access keysbucket_name : strAmazon S3 bucket name that they are trying to accessfile_name : strname of the file that they want to extractReturns:out : DataFrameDataFrame loaded from file in the specfied bucket
Python Code
import pandas as pd
import io
import boto3
import keyring
from flojoy import flojoy, DataFrame


@flojoy
def READ_S3(
    s3_name: str = "",
    bucket_name: str = "",
    file_name: str = "",
) -> DataFrame:
    """The READ_S3 node takes a S3_key name, S3 bucket name, and file name as input, and extracts the file from the specified bucket using the S3_key that was saved.

    Inputs
    ------
    default: None

    Parameters
    ----------
    s3_name : str
        name of the key that the user used to save the access and secret access keys
    bucket_name : str
        Amazon S3 bucket name that they are trying to access
    file_name : str
        name of the file that they want to extract

    Returns
    -------
    DataFrame
        DataFrame loaded from file in the specfied bucket

    """

    if s3_name == "":
        raise ValueError("Provide a name that was used to set AWS S3 key")

    try:
        accessKey = keyring.get_password("system", f"{s3_name}_ACCESSKEY")
        secretKey = keyring.get_password("system", f"{s3_name}_SECRETKEY")
        s3 = boto3.resource(
            "s3", aws_access_key_id=accessKey, aws_secret_access_key=secretKey
        )
        object = s3.Object(bucket_name, file_name)
        buffer = io.BytesIO()
        object.download_fileobj(buffer)
        df = pd.read_parquet(buffer)

        return DataFrame(df=df)

    except Exception as e:
        print(e)

Find this Flojoy Block on GitHub

Example

Having problem with this example app? Join our Discord community and we will help you out!
React Flow mini map

In this example, READ_S3 node extracts a file from AWS S3 bucket and displays it on the Table visualizer.