{"id":6540,"date":"2020-07-22T10:07:25","date_gmt":"2020-07-22T10:07:25","guid":{"rendered":"https:\/\/beta.bluetab.net\/bluetab\/"},"modified":"2023-10-17T12:52:43","modified_gmt":"2023-10-17T12:52:43","slug":"bluetab","status":"publish","type":"post","link":"https:\/\/bluetab.co.uk\/en\/bluetab\/","title":{"rendered":"Basic AWS Glue concepts"},"content":{"rendered":"\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

Basic AWS Glue concepts<\/h1>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t
\"\"<\/a><\/figure>

\u00c1lvaro Santos<\/a><\/h4>

Senior Cloud Solution Architect\u200b<\/p><\/div><\/div>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t

\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t<\/i>\t\t\t\t\t\t\t<\/span>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t<\/i>\t\t\t\t\t\t\t<\/span>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

At\u00a0Cloud Practice<\/strong>\u00a0we aim to encourage adoption of the cloud as a way of working in the IT world. To help with this task, we are going to publish numerous good practice articles and use cases and others will talk about those key services within the cloud.<\/p>

We present the basic concepts\u00a0AWS Glue<\/a><\/strong>\u00a0below.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t
\n\t\t\t
<\/div>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

What is AWS Glue?<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

AWS Glue<\/strong>\u00a0is one of those\u00a0AWS<\/em>\u00a0services that are relatively new but have enormous potential. In particular, this service could be very useful to all those companies that work with data and do not yet have powerful Big Data infrastructure.<\/p>

Basically, Glue is a fully\u00a0AWS<\/em>-managed pay-as-you-go ETL service without the need for provisioning instances. To achieve this, it combines the speed and power of\u00a0Apache Spark<\/em>\u00a0with the data organisation offered by\u00a0Hive Metastore<\/em>.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t
\n\t\t\t
<\/div>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

AWS Glue Data Catalogue<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

The Glue Data Catalogue is where all the data sources and destinations for Glue jobs are stored.<\/p>

  • Table<\/strong>\u00a0is the definition of a metadata table on the data sources and not the data itself.\u00a0AWS Glue<\/em>\u00a0tables can refer to data based on files stored in S3 (such as Parquet, CSV, etc.), RDBMS tables\u2026

    <\/li>
  • Database<\/strong>\u00a0refers to a grouping of data sources to which the tables belong.

    <\/li>
  • Connection<\/strong>\u00a0is a link configured between\u00a0AWS Glue<\/em>\u00a0and\u00a0an RDS<\/em>,\u00a0Redshift<\/em>\u00a0or other\u00a0JDBC<\/em>-compliant database cluster. These allow Glue to access their data.

    <\/li>
  • Crawler<\/strong>\u00a0is the service that connects to a data store, it progresses through a prioritised list of classifiers to determine the schema for the data and to generate the metadata tables. They support determining the schema of complex unstructured or semi-structured data. This is especially important when working with\u00a0Parquet<\/em>,\u00a0AVRO<\/em>, etc. data sources.<\/li><\/ul>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t\t\t
    \n\t\t\t
    <\/div>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t

    ETL<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t

    An\u00a0ETL<\/em>\u00a0in\u00a0AWS Glue<\/em>\u00a0consists primarily of scripts and other tools that use the data configured in the\u00a0Data Catalogue<\/em>\u00a0to extract, transform and load the data into a defined site.<\/p>

    • Job<\/strong>\u00a0is the main ETL engine. A job consists of a script that loads data from the sources defined in the catalogue and performs transformations on them. Glue can generate a script automatically or you can create a customised one using the\u00a0Apache Spark<\/em>\u00a0API in\u00a0Python<\/em>\u00a0(PySpark<\/em>) or\u00a0Scala<\/em>. It also allows the use of external libraries which will be linked to the job by means of a zip file in S3.

      <\/li>
    • Triggers<\/strong>\u00a0are responsible for running the\u00a0Jobs<\/em>. They can be run according to a timetable, a CloudWatch event or even a cron command.

      <\/li>
    • Workflows<\/strong>\u00a0is a set of\u00a0triggers<\/em>,\u00a0crawlers<\/em>\u00a0and\u00a0jobs<\/em>\u00a0related to each other in\u00a0AWS Glue<\/em>. You can use them to create a workflow to perform a complex multi-step ETL, but that\u00a0AWS Glue<\/em>\u00a0can run as a single entity.

      <\/li>
    • ML Transforms<\/strong>\u00a0are specific jobs that use\u00a0Machine Learning<\/em>\u00a0models to create custom transforms for data cleaning, such as identifying duplicate data, for example.

      <\/li>
    • Finally, you can also use\u00a0Dev Endpoints<\/strong>\u00a0and\u00a0Notebooks<\/strong>, which make it faster and easier to develop and test scripts.<\/li><\/ul>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
      \n\t\t\t\t
      \n\t\t\t\t\t\t\t
      \n\t\t\t
      <\/div>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
      \n\t\t\t\t
      \n\t\t\t\t\t

      Examples<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
      \n\t\t\t\t
      \n\t\t\t\t\t\t\t\t\t

      Sample ETL script in Python:<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

      \n\t\t\t\t
      \n\t\t\t\t\t
      import sys\r\nfrom awsglue.transforms import *\r\nfrom awsglue.utils import getResolvedOptions\r\nfrom pyspark.context import SparkContext\r\nfrom awsglue.context import GlueContext\r\nfrom awsglue.dynamicframe import DynamicFrame\r\nfrom awsglue.job import Job\r\n\r\nargs = getResolvedOptions(sys.argv, ['JOB_NAME'])\r\n\r\nsc = SparkContext()\r\n\r\nglueContext = GlueContext(sc)\r\nspark = glueContext.spark_session\r\njob = Job(glueContext)\r\n\r\n## Read Data from a RDS DB using JDBC driver\r\nconnection_option = {\r\n"url": "jdbc:mysql:\/\/mysql–instance1.123456789012.us-east-1.rds.amazonaws.com:3306\/database",\r\n"user": "test",\r\n"password": "password",\r\n"dbtable": "test_table",\r\n"hashexpression": "column_name",\r\n"hashpartitions": "10"\r\n}\r\n\r\nsource_df = glueContext.create_dynamic_frame.from_options('mysql', connection_options = connection_option, transformation_ctx = "source_df")\r\n\r\njob.init(args['JOB_NAME'], args)\r\n\r\n## Convert DataFrames to *AWS Glue* 's DynamicFrames Object\r\ndynamic_df = DynamicFrame.fromDF(source_df, glueContext, "dynamic_df")\r\n\r\n## Write Dynamic Frame to S3 in CSV format\r\ndatasink = glueContext.write_dynamic_frame.from_options(frame = dynamic_df, connection_type = "s3", connection_options = {\r\n"path": "s3:\/\/glueuserdata"\r\n}, format = "csv", transformation_ctx = "datasink")\r\n\r\njob.commit() <\/code><\/pre>