{"type":"doc","content":[{"type":"extension","attrs":{"layout":"default","extensionType":"com.atlassian.confluence.macro.core","extensionKey":"toc","parameters":{"macroParams":{"minLevel":{"value":"1"},"maxLevel":{"value":"2"},"outline":{"value":"false"},"type":{"value":"flat"},"separator":{"value":"pipe"},"printable":{"value":"false"}},"macroMetadata":{"macroId":{"value":"9d0eea93-3d17-41c2-b395-461c9fd47524"},"schemaVersion":{"value":"1"},"title":"Table of Contents"}},"localId":"d8830320-5b0e-4138-b913-0a84ca0ef53c"}},{"type":"heading","attrs":{"level":1},"content":[{"text":"Overview","type":"text"}]},{"type":"paragraph","content":[{"text":"Pathfinder uses ","type":"text"},{"text":"Ceph","type":"text","marks":[{"type":"link","attrs":{"href":"https://docs.ceph.com/en/pacific/rados/index.html"}}]},{"text":" internally to provide a generic S3 compliant storage endpoint. Ceph’s S3 implementation has many user configurable settings that can be tweaked from the CLI. This document aims to give an overview of how to do some less-common operations on S3 buckets. ","type":"text"}]},{"type":"heading","attrs":{"level":1},"content":[{"text":"Editing Cross-Origin Resource Sharing (CORS) Restrictions","type":"text"}]},{"type":"paragraph","content":[{"text":"Cross-Origin Resource Sharing","type":"text","marks":[{"type":"link","attrs":{"href":"https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS"}}]},{"text":" (CORS) is a mechanism that allows a server to indicate other origins from which a browser may permit loading resources. Using S3 with CORS may offer better performance as requests can be made directly from the user to a public facing S3 bucket. By default, all CORS attributes are restricted, as they are a potential security hazard. These attributes can be edited using the CLI tool ","type":"text"},{"text":"s3cmd","type":"text","marks":[{"type":"code"}]},{"text":" on a per-bucket basis. ","type":"text"}]},{"type":"panel","attrs":{"panelType":"note"},"content":[{"type":"paragraph","content":[{"text":"These instructions are to be done on your local workstation. ","type":"text"}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Installing ","type":"text"},{"text":"s3cmd","type":"text","marks":[{"type":"code"}]}]},{"type":"paragraph","content":[{"text":"Make sure you have ","type":"text"},{"text":"s3cmd","type":"text","marks":[{"type":"code"}]},{"text":" installed. If you are using Ubuntu or a RHEL-based Linux distro, you can easily install it via your native package manager. If you are on Windows or Mac, you can install it with ","type":"text"},{"text":"pip","type":"text","marks":[{"type":"code"}]},{"text":". ","type":"text"}]},{"type":"codeBlock","content":[{"text":"# Ubuntu\n$ sudo apt install -y s3cmd\n\n# RHEL-based\n$ sudo dnf install -y s3cmd\n\n# Windows/Mac, you may have to install pip first.\n# Refer to this page: https://pip.pypa.io/en/stable/installation/\n$ pip install --user s3cmd","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Setup","type":"text"}]},{"type":"paragraph","content":[{"text":"Once ","type":"text"},{"text":"s3cmd","type":"text","marks":[{"type":"code"}]},{"text":" is installed, we need to tweak a few initial parameters to support Pathfinder. Run this command to configure the ","type":"text"},{"text":"s3cmd","type":"text","marks":[{"type":"code"}]},{"text":" tool and tweak these values to include your access key and secret key. Make sure to override the default “S3 Endpoint” and “DNS-style bucket+hostname:port template” to match the template below. ","type":"text"}]},{"type":"codeBlock","content":[{"text":"$ s3cmd --configure\n\nEnter new values or accept defaults in brackets with Enter.\nRefer to user manual for detailed description of all options.\n\nAccess key and Secret key are your identifiers for Amazon S3. Leave them empty for using the env variables.\nAccess Key: \nSecret Key: \nDefault Region [US]: US\n\nUse \"s3.amazonaws.com\" for S3 Endpoint and not modify it to the target Amazon S3.\nS3 Endpoint [s3.amazonaws.com]: pathfinder.arcc.uwyo.edu\n\nUse \"%(bucket)s.s3.amazonaws.com\" to the target Amazon S3. \"%(bucket)s\" and \"%(location)s\" vars can be used\nif the target S3 system supports dns based buckets.\nDNS-style bucket+hostname:port template for accessing a bucket [%(bucket)s.s3.amazonaws.com]: %(bucket)s.pathfinder.arcc.uwyo.edu\n\nEncryption password is used to protect your files from reading\nby unauthorized persons while in transfer to S3\nEncryption password:\nPath to GPG program [/usr/bin/gpg]:\n\nWhen using secure HTTPS protocol all communication with Amazon S3\nservers is protected from 3rd party eavesdropping. This method is\nslower than plain HTTP, and can only be proxied with Python 2.7 or newer\nUse HTTPS protocol [Yes]: Yes\n\nOn some networks all internet access must go through a HTTP proxy.\nTry setting it here if you can't connect to S3 directly\nHTTP Proxy server name:\n\nNew settings:\n Access Key: \n Secret Key: \n Default Region: US\n S3 Endpoint: pathfinder.arcc.uwyo.edu\n DNS-style bucket+hostname:port template for accessing a bucket: %(bucket)s.pathfinder.arcc.uwyo.edu\n Encryption password:\n Path to GPG program: /usr/bin/gpg\n Use HTTPS protocol: True\n HTTP Proxy server name:\n HTTP Proxy server port: 0\n\nTest access with supplied credentials? [Y/n] n","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Testing ","type":"text"},{"text":"s3cmd","type":"text","marks":[{"type":"code"}]}]},{"type":"paragraph","content":[{"text":"Once ","type":"text"},{"text":"s3cmd","type":"text","marks":[{"type":"code"}]},{"text":" is installed, we can try querying your buckets to make sure everything works correctly.","type":"text"}]},{"type":"codeBlock","content":[{"text":"$ s3cmd ls\n2023-11-10 14:39 s3://","type":"text"}]},{"type":"paragraph","content":[{"text":"If you see your corresponding buckets, ","type":"text"},{"text":"s3cmd","type":"text","marks":[{"type":"code"}]},{"text":" is configured correctly. ","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Setting CORS Attributes","type":"text"}]},{"type":"paragraph","content":[{"text":"Once ","type":"text"},{"text":"s3cmd","type":"text","marks":[{"type":"code"}]},{"text":" is installed and configured, we can proceed with editing the CORS attributes on your bucket. ","type":"text"}]},{"type":"paragraph","content":[{"text":"Create an XML file with your CORS rules. For example, let’s save this as ","type":"text"},{"text":"cors.xml","type":"text","marks":[{"type":"code"}]},{"text":":","type":"text"}]},{"type":"codeBlock","attrs":{"language":"xml"},"content":[{"text":"\n \n http://uwyo.edu\n PUT\n POST\n DELETE\n *\n 3000\n \n","type":"text"}]},{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"paragraph","content":[{"text":"More details about the XML format for CORS attributes is available here: ","type":"text"},{"type":"inlineCard","attrs":{"url":"https://docs.aws.amazon.com/AmazonS3/latest/userguide/ManageCorsUsing.html"}},{"text":" ","type":"text"}]},{"type":"paragraph","content":[{"text":"Note: Pathfinder only supports using XML, not the newer JSON format. ","type":"text"}]}]},{"type":"paragraph","content":[{"text":"Apply the newly created XML CORS definition to your bucket. Using the ","type":"text"},{"text":"setcors","type":"text","marks":[{"type":"code"}]},{"text":" sub-command will overwrite any existing CORS rules, if you have defined them in the past. ","type":"text"}]},{"type":"codeBlock","content":[{"text":"$ s3cmd setcors /path/to/cors.xml s3://","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Removing CORS Attributes","type":"text"}]},{"type":"paragraph","content":[{"text":"To remove all CORS attributes from a bucket, use the ","type":"text"},{"text":"delcors","type":"text","marks":[{"type":"code"}]},{"text":" sub-command.","type":"text"}]},{"type":"codeBlock","content":[{"text":"$ s3cmd delcors s3://","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Caveats","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Pathfinder only supports the XML format of CORS attributes. The newer JSON format is not yet supported.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Setting CORS attributes will overwrite any existing attributes.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"There is currently no way to query existing CORS attributes from a given bucket. ","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Setting attributes must be done on a per-bucket basis, there is no way to set CORS rules across all buckets owned by a particular user. ","type":"text"}]}]}]},{"type":"paragraph"}],"version":1}

Browser not supported