Wer kennt sich mit Selenium aus oder hat zufällig ein geeignetes Script?

**unravelmutton** · 29.08.2024, 03:21

Zitat von BlueC Beitrag anzeigen

Morning,
So let's assume I want to read certain metadata from the source code of my website at regular intervals using Selenium...
To do this, a ready-made PHP script (+cron job?) should be started on my local system that can take on this job.
There is a list of sub-URLs as a text file that are called up one after the other and processed by this script.
The whole thing should work in such a way that one URL at a time from this list is processed,
the result is temporarily stored in a text file and Selenium only continues to work after the parsing is finished.

Or to put it graphically:

Selenium runs the browser > URL list is loaded > URL1 > PHP script in path X is executed for this URL > source code is read out > echo in .txt file > continue with URL2...
URL list is loaded > URL2 > PHP script in path X is executed for this URL > source code is read out > echo in .txt file > continue with URL3...
URL list is loaded > URL3 > PHP script in path X is executed for this URL > source code is read out > echo in .txt file > continue with URL4...
etc. etc.
The whole thing should be carried out at regular intervals, e.g. every 24 hours. The number of URLs that need to be processed is relatively high.

(If there is another, simpler solution for this that would be fine too, but I've heard that Selenium is the best and fastest alternative for such browser-based operations?)

Any tips or ideas on how I can best set something like this up?

snow rider

Install Selenium and WebDriver: Install Selenium in the desired programming language (e.g. Python) and download the appropriate WebDriver (e.g. ChromeDriver for Google Chrome).
Create a URL list: This list can be a simple text file containing each URL on a new line.

The PHP script will be responsible for reading the source code of the web page and extracting the relevant metadata. It could look something like this:
<?php
function extractMetadata($url) {
// Load the URL
$html = file_get_contents($url);

// Extract metadata (e.g. <meta> tags)
$doc = new DOMDocument();
@$doc->loadHTML($html);
$metas = $doc->getElementsByTagName('meta');

$data = [];

foreach ($metas as $meta) {
if ($meta->getAttribute('name')) {
$data[$meta->getAttribute('name')] = $meta->getAttribute('content');
}
}

return $data;
}

// Example call
$url = $argv[1];
$metadata = extractMetadata($url);

// Write results to a file
file_put_contents('metadata.txt', print_r($metadata, true), FILE_APPEND);
The Selenium script can be written in Python and should control the browser to open the URLs and execute the PHP script for each URL.
from selenium import webdriver
import time
import subprocess

# WebDriver Setup
driver = webdriver.Chrome('/path/to/chromedriver')

# Load URL list
with open('urls.txt', 'r') as file:
urls = file.readlines()

for url in urls:
url = url.strip()
driver.get(url)

# Run PHP script and pass URL as parameter
subprocess.call(['php', '/path/to/script.php', url])

# Cache results
time.sleep(2) # Wait time if necessary

# Close browser
driver.quit()
Set up a cronjob that runs the Selenium script periodically. Here is an example of a cron job that runs the script every 24 hours:
0 0 * * * /usr/bin/python3 /path/to/selenium_script.py
Alternative solution without Selenium

If you are looking for a simpler and possibly more efficient solution, you can consider processing the URLs directly in PHP or another server-side script without going through Selenium. The PHP script could load the URLs in the list itself, extract metadata and store it. This would probably be faster and more resource-efficient as no browser rendering is required .

**magentaviking** · 28.10.2024, 10:02

Lese den Quelltext mithilfe von Selenium aus und speichere die gewünschten Metadaten in einer .txt-Datei. Achte darauf, dass das PHP-Skript für jede URL den Quelltext speichert, bevor die nächste URL aufgerufen wird.

**jackjohn** · 06.11.2024, 04:05

Moin!

Für dein Vorhaben mit Selenium und PHP kannst du ein Skript erstellen, das die URLs aus einer Textdatei liest und dann nacheinander abarbeitet. Hier ist ein grober Ablauf:

Selenium Setup: Installiere Selenium für PHP (z.B. mit Composer).
Textdatei einlesen: Lade die Liste der URLs in ein Array.
Loop durch URLs: Iteriere über das Array, öffne jede URL mit Selenium.
PHP-Skript ausführen: Rufe dein PHP-Skript auf, um die Metadaten auszulesen.
Ergebnisse speichern: Speichere die Ergebnisse in einer Textdatei.
Cronjob einrichten: Plane das Skript mit Cron, um es alle 24 Stunden auszuführen.

Hier ein einfaches Beispiel für den PHP-Code:
php
Copy
<?php require 'vendor/autoload.php'; // Selenium PHP Client use Facebook\WebDriver\Remote\RemoteWebDriver; use Facebook\WebDriver\Remote\DesiredCapabilities; // Selenium Server URL $serverUrl = 'http://localhost:4444/wd/hub'; $driver = RemoteWebDriver::create($serverUrl, DesiredCapabilities::chrome()); // URLs aus der Datei lesen $urls = file('urls.txt', FILE_IGNORE_NEW_LINES); foreach ($urls as $url) { $driver->get($url); // Hier dein PHP-Skript aufrufen und Metadaten auslesen // Ergebnisse in eine Textdatei speichern } $driver->quit(); ?>
Stelle sicher, dass der Selenium-Server läuft und dein PHP-Skript die Metadaten korrekt verarbeitet. Viel Erfolg!

Wer kennt sich mit Selenium aus oder hat zufällig ein geeignetes Script?